Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:58:44.125711 398 test_util.cc:268] Using random seed: 255707112
I20240503 01:58:44.133106 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:44.133191 398 ts_itest-base.cc:113] --------------
I20240503 01:58:44.133212 398 ts_itest-base.cc:114] 6 tablet servers
I20240503 01:58:44.133231 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:44.133245 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:44Z Disabled control of system clock
I20240503 01:58:44.143286 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34331
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34331
--location_mapping_cmd=/tmp/dist-test-task8Tfalf/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:58:44.248651 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.248900 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.248971 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.252035 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.252112 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:44.252142 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.252180 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:44.252218 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:44.256021 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task8Tfalf/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34331
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.257014 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.258250 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:58:44.266948 410 server_base.cc:1034] running on GCE node
W20240503 01:58:44.268021 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.269071 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.269865 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.270519 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.271569 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.272728 410 hybrid_clock.cc:648] HybridClock initialized: now 1714701524272662 us; error 84 us; skew 500 ppm
I20240503 01:58:44.274814 410 webserver.cc:460] Webserver started at http://127.0.99.190:39873/ using document root <none> and password file <none>
I20240503 01:58:44.275400 410 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.275456 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.275653 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.277436 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "48d6008349a34c919262a7c21e448fff"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.277942 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "48d6008349a34c919262a7c21e448fff"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.281634 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 01:58:44.283984 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.285041 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:44.285172 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "48d6008349a34c919262a7c21e448fff"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.285285 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.300751 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.301440 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.301615 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.308060 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34331
I20240503 01:58:44.308075 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34331 every 8 connection(s)
I20240503 01:58:44.309111 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:44.311842 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:44.317126 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff: Bootstrap starting.
I20240503 01:58:44.319710 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:44.319763 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 410
I20240503 01:58:44.319924 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:44.320600 477 log.cc:824] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:44.322955 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff: No bootstrap required, opened a new log
I20240503 01:58:44.326170 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } }
I20240503 01:58:44.326365 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:44.326400 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d6008349a34c919262a7c21e448fff, State: Initialized, Role: FOLLOWER
I20240503 01:58:44.326892 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } }
I20240503 01:58:44.327044 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:44.327108 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:44.327227 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:44.328228 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } }
I20240503 01:58:44.328583 477 leader_election.cc:302] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48d6008349a34c919262a7c21e448fff; no voters:
I20240503 01:58:44.328903 477 leader_election.cc:288] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:44.329005 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:44.329586 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [term 1 LEADER]: Becoming Leader. State: Replica: 48d6008349a34c919262a7c21e448fff, State: Running, Role: LEADER
I20240503 01:58:44.329922 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } }
I20240503 01:58:44.330070 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:44.331043 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48d6008349a34c919262a7c21e448fff. Latest consensus state: current_term: 1 leader_uuid: "48d6008349a34c919262a7c21e448fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } } }
I20240503 01:58:44.331256 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:44.331367 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48d6008349a34c919262a7c21e448fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d6008349a34c919262a7c21e448fff" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34331 } } }
I20240503 01:58:44.331446 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:44.331709 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:44.334977 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:44.340210 491 catalog_manager.cc:1347] Generated new cluster ID: 36fbc6e38c7549c8879ef337654c8067
I20240503 01:58:44.340291 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:44.365892 491 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:44.366866 491 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:44.377897 491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 48d6008349a34c919262a7c21e448fff: Generated new TSK 0
I20240503 01:58:44.378633 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:44.384614 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:44.494387 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.494632 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.494673 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.497776 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.497867 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.497973 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:44.501750 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.502763 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.504066 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:44.513145 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.513811 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.514964 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.515997 501 server_base.cc:1034] running on GCE node
I20240503 01:58:44.516496 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.517154 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.518332 501 hybrid_clock.cc:648] HybridClock initialized: now 1714701524518305 us; error 43 us; skew 500 ppm
I20240503 01:58:44.520540 501 webserver.cc:460] Webserver started at http://127.0.99.129:41327/ using document root <none> and password file <none>
I20240503 01:58:44.521183 501 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.521265 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.521486 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.523339 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "158b7d66c1e8460aae25035f17afdb14"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.523936 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "158b7d66c1e8460aae25035f17afdb14"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.527637 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 01:58:44.530004 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.531152 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:58:44.531273 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "158b7d66c1e8460aae25035f17afdb14"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.531397 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.560938 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.561630 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.561784 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.562348 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:44.563297 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:44.563354 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.563405 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:44.563426 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.572615 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43283
I20240503 01:58:44.572634 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43283 every 8 connection(s)
I20240503 01:58:44.573688 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:44.580845 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 501
I20240503 01:58:44.580950 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:44.582706 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:44.582988 629 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:44.583218 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:44.583701 629 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:44.620065 441 ts_manager.cc:194] Registered new tserver with Master: 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:44.621387 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:42765
W20240503 01:58:44.697167 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.697427 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.697469 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.700640 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.700717 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.700801 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:44.704792 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.705782 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.706957 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:44.715943 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.716929 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.717828 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.718689 632 server_base.cc:1034] running on GCE node
I20240503 01:58:44.719131 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.719810 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.720961 632 hybrid_clock.cc:648] HybridClock initialized: now 1714701524720948 us; error 30 us; skew 500 ppm
I20240503 01:58:44.722880 632 webserver.cc:460] Webserver started at http://127.0.99.130:37399/ using document root <none> and password file <none>
I20240503 01:58:44.723461 632 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.723520 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.723727 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.725468 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "123bf566427645e9b94b1c01ebf9cd4c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.725956 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "123bf566427645e9b94b1c01ebf9cd4c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.729440 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:44.731830 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.732894 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:58:44.733026 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "123bf566427645e9b94b1c01ebf9cd4c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.733119 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.749594 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.750185 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.750326 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.750836 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:44.751662 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:44.751708 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.751763 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:44.751791 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.760342 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43979
I20240503 01:58:44.760370 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43979 every 8 connection(s)
I20240503 01:58:44.761338 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:44.770234 761 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:44.770483 761 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:44.771049 761 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:44.771319 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 632
I20240503 01:58:44.771414 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:44.775619 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:44.807082 441 ts_manager.cc:194] Registered new tserver with Master: 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:44.807884 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:37393
W20240503 01:58:44.891710 765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:44.891975 765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:44.892041 765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:44.894743 765 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:44.894825 765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:44.894915 765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:44.898097 765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:44.898973 765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:44.900125 765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:44.908628 773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.909559 770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:44.910473 771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:44.911288 765 server_base.cc:1034] running on GCE node
I20240503 01:58:44.911737 765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:44.912393 765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:44.913548 765 hybrid_clock.cc:648] HybridClock initialized: now 1714701524913532 us; error 36 us; skew 500 ppm
I20240503 01:58:44.915385 765 webserver.cc:460] Webserver started at http://127.0.99.131:45661/ using document root <none> and password file <none>
I20240503 01:58:44.916002 765 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:44.916080 765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:44.916265 765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:44.918004 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cf77ccf223a649f590c7c2a7b632768c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.918514 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cf77ccf223a649f590c7c2a7b632768c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.922976 765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240503 01:58:44.926203 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.927368 765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:44.927495 765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cf77ccf223a649f590c7c2a7b632768c"
format_stamp: "Formatted at 2024-05-03 01:58:44 on dist-test-slave-pvfz"
I20240503 01:58:44.927636 765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:44.944226 765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:44.944955 765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:44.945120 765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:44.945698 765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:44.946666 765 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:44.946723 765 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.946786 765 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:44.946816 765 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:44.956575 765 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41289
I20240503 01:58:44.956590 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41289 every 8 connection(s)
I20240503 01:58:44.957605 765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:44.963743 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 765
I20240503 01:58:44.963835 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:44.965977 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:44.966804 893 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:44.967054 893 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:44.967710 893 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:45.002063 441 ts_manager.cc:194] Registered new tserver with Master: cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:45.003049 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55485
W20240503 01:58:45.076974 896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.077217 896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.077258 896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.080471 896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:45.080557 896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.080627 896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:45.084410 896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.085381 896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.086503 896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.095506 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.096100 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.097193 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.098161 896 server_base.cc:1034] running on GCE node
I20240503 01:58:45.098603 896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.099222 896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.100374 896 hybrid_clock.cc:648] HybridClock initialized: now 1714701525100356 us; error 40 us; skew 500 ppm
I20240503 01:58:45.106897 896 webserver.cc:460] Webserver started at http://127.0.99.132:38901/ using document root <none> and password file <none>
I20240503 01:58:45.107522 896 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.107630 896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.107864 896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.109655 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4f8d9686ff2d42428d81510426857a12"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.110224 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4f8d9686ff2d42428d81510426857a12"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.114827 896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:58:45.117208 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.118228 896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:45.118336 896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4f8d9686ff2d42428d81510426857a12"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.118422 896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.137995 896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.138698 896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.138904 896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.139482 896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:45.140439 896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:45.140489 896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.140552 896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:45.140578 896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.150565 896 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40495
I20240503 01:58:45.150591 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40495 every 8 connection(s)
I20240503 01:58:45.151587 896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:45.153741 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 896
I20240503 01:58:45.153846 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:45.156019 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:45.160861 1025 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:45.161091 1025 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:45.161597 1025 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:45.207695 441 ts_manager.cc:194] Registered new tserver with Master: 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:45.208392 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:58649
W20240503 01:58:45.276381 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.276636 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.276707 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.279843 1028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:45.279924 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.280037 1028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:45.283609 1028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.1028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.284459 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.285555 1028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.294128 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.295100 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.296033 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.296862 1028 server_base.cc:1034] running on GCE node
I20240503 01:58:45.297256 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.297814 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.298988 1028 hybrid_clock.cc:648] HybridClock initialized: now 1714701525298980 us; error 32 us; skew 500 ppm
I20240503 01:58:45.300752 1028 webserver.cc:460] Webserver started at http://127.0.99.133:34277/ using document root <none> and password file <none>
I20240503 01:58:45.301283 1028 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.301330 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.301476 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.303112 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "24a60c6be59b44b88af9c886a1b1ef57"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.303565 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "24a60c6be59b44b88af9c886a1b1ef57"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.306751 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:45.309019 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.310081 1028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:45.310190 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "24a60c6be59b44b88af9c886a1b1ef57"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.310293 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.329820 1028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.330600 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.330763 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.331396 1028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:45.332404 1028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:45.332453 1028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.332497 1028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:45.332535 1028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.341480 1028 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37453
I20240503 01:58:45.341503 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37453 every 8 connection(s)
I20240503 01:58:45.342447 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:45.350579 1157 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:45.350771 1157 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:45.351258 1157 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:45.351919 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 1028
I20240503 01:58:45.352005 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:45.356331 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--builtin_ntp_servers=127.0.99.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:45.385214 441 ts_manager.cc:194] Registered new tserver with Master: 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:45.385859 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:34821
W20240503 01:58:45.462016 1161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.462249 1161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.462324 1161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.465204 1161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:45.465268 1161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.465364 1161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:58:45.468737 1161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34331
--never_fsync=true
--heap_profile_path=/tmp/kudu.1161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.469609 1161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.470669 1161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.479260 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.479748 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.480798 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.481675 1161 server_base.cc:1034] running on GCE node
I20240503 01:58:45.482111 1161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.482688 1161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.483834 1161 hybrid_clock.cc:648] HybridClock initialized: now 1714701525483823 us; error 29 us; skew 500 ppm
I20240503 01:58:45.485607 1161 webserver.cc:460] Webserver started at http://127.0.99.134:42069/ using document root <none> and password file <none>
I20240503 01:58:45.486150 1161 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.486209 1161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.486387 1161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.488035 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.488461 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.491729 1161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:45.493832 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.494827 1161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:45.494941 1161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-pvfz"
I20240503 01:58:45.495019 1161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.508646 1161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.509375 1161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.509541 1161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.510105 1161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:45.511077 1161 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:45.511128 1161 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.511198 1161 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:45.511224 1161 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.520826 1288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37471 every 8 connection(s)
I20240503 01:58:45.520803 1161 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37471
I20240503 01:58:45.521885 1161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:58:45.523182 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 1161
I20240503 01:58:45.523285 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:58:45.531006 1289 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34331
I20240503 01:58:45.531229 1289 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:45.531765 1289 heartbeater.cc:511] Master 127.0.99.190:34331 requested a full tablet report, sending...
I20240503 01:58:45.565421 441 ts_manager.cc:194] Registered new tserver with Master: b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:45.566071 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34769
I20240503 01:58:45.573237 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 01:58:45.584887 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 01:58:45.624328 629 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:45.810464 761 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:46.005452 893 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:46.210609 1025 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:46.388262 1157 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:46.568439 1289 heartbeater.cc:503] Master 127.0.99.190:34331 was elected leader, sending a full tablet report...
I20240503 01:58:49.337494 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:49.359283 562 tablet_service.cc:1467] Processing CreateTablet for tablet d2a5bae6b9644d68bec77be04d9d3c45 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.359391 959 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf27fda05442ed9fcb87769517b9fe (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.359808 563 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf27fda05442ed9fcb87769517b9fe (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.359283 695 tablet_service.cc:1467] Processing CreateTablet for tablet 84225e096ac143ebb59cffd163f2d151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.360302 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a5bae6b9644d68bec77be04d9d3c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.360303 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcf27fda05442ed9fcb87769517b9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.360424 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84225e096ac143ebb59cffd163f2d151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.363821 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcf27fda05442ed9fcb87769517b9fe (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.364189 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcf27fda05442ed9fcb87769517b9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.364720 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcf27fda05442ed9fcb87769517b9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.366191 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 84225e096ac143ebb59cffd163f2d151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.367036 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84225e096ac143ebb59cffd163f2d151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.368722 1314 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:49.368804 1090 tablet_service.cc:1467] Processing CreateTablet for tablet d2a5bae6b9644d68bec77be04d9d3c45 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.369047 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a5bae6b9644d68bec77be04d9d3c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.371134 1314 tablet_bootstrap.cc:654] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.371974 1314 log.cc:824] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.373739 1314 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:49.373920 1314 ts_tablet_manager.cc:1367] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:49.377260 826 tablet_service.cc:1467] Processing CreateTablet for tablet d2a5bae6b9644d68bec77be04d9d3c45 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.377815 1314 raft_consensus.cc:348] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.377992 1318 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:49.378027 1314 raft_consensus.cc:374] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.378077 1314 raft_consensus.cc:729] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.378225 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a5bae6b9644d68bec77be04d9d3c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.378605 827 tablet_service.cc:1467] Processing CreateTablet for tablet 84225e096ac143ebb59cffd163f2d151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d89f4f9675d44d2a0de9ecacc550353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.378866 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84225e096ac143ebb59cffd163f2d151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.380300 1314 consensus_queue.cc:260] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.380877 1316 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:49.381362 1314 ts_tablet_manager.cc:1398] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 01:58:49.381404 1318 tablet_bootstrap.cc:654] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.381752 1314 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:49.382251 1318 log.cc:824] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.382692 1314 tablet_bootstrap.cc:654] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.382968 1319 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:49.382678 1317 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:49.383780 1316 tablet_bootstrap.cc:654] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.384588 1316 log.cc:824] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.385075 1317 tablet_bootstrap.cc:654] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.385075 1319 tablet_bootstrap.cc:654] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.385825 1319 log.cc:824] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.385825 1317 log.cc:824] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.387418 1323 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:49.387511 1319 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:49.387516 1317 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:49.387717 1317 ts_tablet_manager.cc:1367] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:49.387717 1319 ts_tablet_manager.cc:1367] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:49.389499 1323 tablet_bootstrap.cc:654] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.390612 1323 log.cc:824] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.392318 1317 raft_consensus.cc:348] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.392567 1317 raft_consensus.cc:374] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.392627 1317 raft_consensus.cc:729] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.392741 1314 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:49.392835 1314 ts_tablet_manager.cc:1367] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:58:49.393150 1317 consensus_queue.cc:260] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.393452 1318 tablet_bootstrap.cc:492] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:49.393415 1314 raft_consensus.cc:348] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.393523 1314 raft_consensus.cc:374] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.393551 1314 raft_consensus.cc:729] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.393640 1318 ts_tablet_manager.cc:1367] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 01:58:49.393680 1314 consensus_queue.cc:260] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.393913 1314 ts_tablet_manager.cc:1398] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:58:49.394112 1317 ts_tablet_manager.cc:1398] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:49.395185 1316 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:49.395361 1316 ts_tablet_manager.cc:1367] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 01:58:49.395821 1319 raft_consensus.cc:348] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.396052 1319 raft_consensus.cc:374] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.396122 1319 raft_consensus.cc:729] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.397481 1319 consensus_queue.cc:260] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.398586 1319 ts_tablet_manager.cc:1398] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 01:58:49.399375 1323 tablet_bootstrap.cc:492] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:49.399657 1323 ts_tablet_manager.cc:1367] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 01:58:49.400044 1316 raft_consensus.cc:348] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.400295 1316 raft_consensus.cc:374] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.400369 1316 raft_consensus.cc:729] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.400835 1316 consensus_queue.cc:260] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.400913 1318 raft_consensus.cc:348] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.401129 1318 raft_consensus.cc:374] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.401185 1318 raft_consensus.cc:729] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.401609 1318 consensus_queue.cc:260] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.401873 1316 ts_tablet_manager.cc:1398] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:49.402256 1316 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:49.402554 1318 ts_tablet_manager.cc:1398] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 01:58:49.403401 1316 tablet_bootstrap.cc:654] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.404515 1323 raft_consensus.cc:348] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.404796 1316 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:49.404824 1323 raft_consensus.cc:374] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.404891 1316 ts_tablet_manager.cc:1367] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:49.404973 1323 raft_consensus.cc:729] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.405423 1316 raft_consensus.cc:348] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.405557 1316 raft_consensus.cc:374] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.405594 1316 raft_consensus.cc:729] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.405506 1323 consensus_queue.cc:260] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.405723 1316 consensus_queue.cc:260] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
W20240503 01:58:49.405918 1026 tablet.cc:2369] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:49.406049 1316 ts_tablet_manager.cc:1398] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:49.407318 1323 ts_tablet_manager.cc:1398] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 01:58:49.407598 1323 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:49.408993 1323 tablet_bootstrap.cc:654] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.410226 1323 tablet_bootstrap.cc:492] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:49.410307 1323 ts_tablet_manager.cc:1367] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:49.410866 1323 raft_consensus.cc:348] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.410962 1323 raft_consensus.cc:374] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.410984 1323 raft_consensus.cc:729] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.411101 1323 consensus_queue.cc:260] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.411342 1323 ts_tablet_manager.cc:1398] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:49.416925 1333 raft_consensus.cc:482] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.417124 1333 raft_consensus.cc:504] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.418229 1333 leader_election.cc:288] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:49.422883 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a5bae6b9644d68bec77be04d9d3c45" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:49.422955 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a5bae6b9644d68bec77be04d9d3c45" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:49.423213 583 raft_consensus.cc:2454] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:49.423213 1111 raft_consensus.cc:2454] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:49.423686 781 leader_election.cc:302] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:49.423942 1333 raft_consensus.cc:2781] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.424023 1333 raft_consensus.cc:482] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.424069 1333 raft_consensus.cc:3037] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.430938 1333 raft_consensus.cc:504] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.431483 1333 leader_election.cc:288] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:49.431984 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a5bae6b9644d68bec77be04d9d3c45" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:49.431993 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a5bae6b9644d68bec77be04d9d3c45" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:49.432114 1111 raft_consensus.cc:3037] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.432137 583 raft_consensus.cc:3037] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.437355 1111 raft_consensus.cc:2454] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:49.437373 583 raft_consensus.cc:2454] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:49.437736 781 leader_election.cc:302] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:49.437964 1333 raft_consensus.cc:2781] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.438216 1333 raft_consensus.cc:686] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [term 1 LEADER]: Becoming Leader. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Running, Role: LEADER
I20240503 01:58:49.438603 1333 consensus_queue.cc:237] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.440838 1331 raft_consensus.cc:482] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.441020 1331 raft_consensus.cc:504] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.442296 1331 leader_election.cc:288] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:49.442371 441 catalog_manager.cc:5484] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf77ccf223a649f590c7c2a7b632768c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "cf77ccf223a649f590c7c2a7b632768c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:49.446691 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcf27fda05442ed9fcb87769517b9fe" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:49.446993 583 raft_consensus.cc:2454] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:49.447384 1179 leader_election.cc:302] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.447685 1331 raft_consensus.cc:2781] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.447765 1331 raft_consensus.cc:482] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.447818 1331 raft_consensus.cc:3037] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.447890 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcf27fda05442ed9fcb87769517b9fe" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:49.448158 979 raft_consensus.cc:2454] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:49.452814 1331 raft_consensus.cc:504] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.453223 1331 leader_election.cc:288] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:49.453622 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcf27fda05442ed9fcb87769517b9fe" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:49.453632 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcf27fda05442ed9fcb87769517b9fe" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:49.453739 583 raft_consensus.cc:3037] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.453765 979 raft_consensus.cc:3037] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.458699 583 raft_consensus.cc:2454] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:49.458688 979 raft_consensus.cc:2454] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:49.459105 1179 leader_election.cc:302] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.459358 1331 raft_consensus.cc:2781] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.459640 1331 raft_consensus.cc:686] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 LEADER]: Becoming Leader. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Running, Role: LEADER
I20240503 01:58:49.460000 1331 consensus_queue.cc:237] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
W20240503 01:58:49.461947 894 tablet.cc:2369] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:49.463130 441 catalog_manager.cc:5484] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d reported cstate change: term changed from 0 to 1, leader changed from <none> to b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:49.467180 1328 raft_consensus.cc:482] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.467370 1328 raft_consensus.cc:504] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.468868 1328 leader_election.cc:288] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:49.472929 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84225e096ac143ebb59cffd163f2d151" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:49.473075 1111 raft_consensus.cc:2454] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:49.473426 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84225e096ac143ebb59cffd163f2d151" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:49.473534 649 leader_election.cc:302] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:49.473623 847 raft_consensus.cc:2454] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:49.473810 1328 raft_consensus.cc:2781] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.473896 1328 raft_consensus.cc:482] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.473946 1328 raft_consensus.cc:3037] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.478650 1328 raft_consensus.cc:504] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.479038 1328 leader_election.cc:288] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:49.479439 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84225e096ac143ebb59cffd163f2d151" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:49.479473 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84225e096ac143ebb59cffd163f2d151" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:49.479596 1111 raft_consensus.cc:3037] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.479629 847 raft_consensus.cc:3037] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.511009 847 raft_consensus.cc:2454] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:49.511009 1111 raft_consensus.cc:2454] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:49.511440 649 leader_election.cc:302] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:49.511674 1328 raft_consensus.cc:2781] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.511948 1328 raft_consensus.cc:686] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:49.512327 1328 consensus_queue.cc:237] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:49.515403 441 catalog_manager.cc:5484] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:49.516006 762 tablet.cc:2369] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:49.525862 1290 tablet.cc:2369] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:49.578379 630 tablet.cc:2369] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:49.596532 1158 tablet.cc:2369] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:49.671531 1347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:49.671872 1347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:49.675200 1347 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:58:49.786037 847 raft_consensus.cc:1261] T 84225e096ac143ebb59cffd163f2d151 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.786373 1111 raft_consensus.cc:1261] T 84225e096ac143ebb59cffd163f2d151 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.786659 1328 consensus_queue.cc:1035] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.786964 1345 consensus_queue.cc:1035] T 84225e096ac143ebb59cffd163f2d151 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.788048 1111 raft_consensus.cc:1261] T d2a5bae6b9644d68bec77be04d9d3c45 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.788152 583 raft_consensus.cc:1261] T d2a5bae6b9644d68bec77be04d9d3c45 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.788450 1333 consensus_queue.cc:1035] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.789525 1333 consensus_queue.cc:1035] T d2a5bae6b9644d68bec77be04d9d3c45 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.796262 1375 mvcc.cc:205] Tried to move back new op lower bound from 7023417466005082112 to 7023417464584056832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417466005082112}]
I20240503 01:58:49.798162 979 raft_consensus.cc:1261] T 9fcf27fda05442ed9fcb87769517b9fe P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.798386 583 raft_consensus.cc:1261] T 9fcf27fda05442ed9fcb87769517b9fe P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:49.798898 1331 consensus_queue.cc:1035] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.799319 1331 consensus_queue.cc:1035] T 9fcf27fda05442ed9fcb87769517b9fe P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.807655 1389 mvcc.cc:205] Tried to move back new op lower bound from 7023417466005082112 to 7023417464584056832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417466005082112}]
I20240503 01:58:49.811682 1381 mvcc.cc:205] Tried to move back new op lower bound from 7023417465975975936 to 7023417464885157888. Current Snapshot: MvccSnapshot[applied={T|T < 7023417465975975936}]
I20240503 01:58:49.813328 1373 mvcc.cc:205] Tried to move back new op lower bound from 7023417465975975936 to 7023417464885157888. Current Snapshot: MvccSnapshot[applied={T|T < 7023417465975975936}]
I20240503 01:58:49.818112 1372 mvcc.cc:205] Tried to move back new op lower bound from 7023417465996980224 to 7023417464671076352. Current Snapshot: MvccSnapshot[applied={T|T < 7023417465996980224}]
I20240503 01:58:49.840827 1388 mvcc.cc:205] Tried to move back new op lower bound from 7023417465996980224 to 7023417464671076352. Current Snapshot: MvccSnapshot[applied={T|T < 7023417465996980224}]
INSERT report
rows total: 2000
time total: 152.274 ms
time per row: 0.0761368 ms
I20240503 01:58:49.871667 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:49.877091 959 tablet_service.cc:1467] Processing CreateTablet for tablet edace0c39397435c929e99df6f2a5f0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.877209 1223 tablet_service.cc:1467] Processing CreateTablet for tablet edace0c39397435c929e99df6f2a5f0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.877439 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edace0c39397435c929e99df6f2a5f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.877434 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edace0c39397435c929e99df6f2a5f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.877903 694 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2aaee1132c40638fae59ad5f985056 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.878201 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2aaee1132c40638fae59ad5f985056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.878530 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 57d0b13b23c7421ea802353cbaad45a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.878775 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d0b13b23c7421ea802353cbaad45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.878964 695 tablet_service.cc:1467] Processing CreateTablet for tablet edace0c39397435c929e99df6f2a5f0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.878988 562 tablet_service.cc:1467] Processing CreateTablet for tablet 57d0b13b23c7421ea802353cbaad45a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.879182 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edace0c39397435c929e99df6f2a5f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.879225 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d0b13b23c7421ea802353cbaad45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.880592 1317 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:49.880857 1316 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:49.881240 1314 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:49.881489 1317 tablet_bootstrap.cc:654] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.881675 1316 tablet_bootstrap.cc:654] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.881747 958 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2aaee1132c40638fae59ad5f985056 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.882078 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2aaee1132c40638fae59ad5f985056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.882103 1314 tablet_bootstrap.cc:654] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.882912 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2aaee1132c40638fae59ad5f985056 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.883081 827 tablet_service.cc:1467] Processing CreateTablet for tablet 57d0b13b23c7421ea802353cbaad45a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=aea55467b56c4849aed10dcd5216c7f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:49.883188 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2aaee1132c40638fae59ad5f985056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.883199 1316 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:49.883277 1316 ts_tablet_manager.cc:1367] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:49.883325 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d0b13b23c7421ea802353cbaad45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.883812 1316 raft_consensus.cc:348] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.883919 1316 raft_consensus.cc:374] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.883947 1316 raft_consensus.cc:729] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.884063 1316 consensus_queue.cc:260] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.884378 1316 ts_tablet_manager.cc:1398] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.885465 1319 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:49.886256 1323 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:49.886332 1314 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:49.886410 1314 ts_tablet_manager.cc:1367] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:49.886521 1319 tablet_bootstrap.cc:654] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.886904 1314 raft_consensus.cc:348] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.887015 1314 raft_consensus.cc:374] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.887055 1314 raft_consensus.cc:729] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.887137 1323 tablet_bootstrap.cc:654] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.887209 1314 consensus_queue.cc:260] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.887534 1314 ts_tablet_manager.cc:1398] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.887786 1318 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:49.888520 1318 tablet_bootstrap.cc:654] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.889726 1318 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:49.889815 1318 ts_tablet_manager.cc:1367] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:49.890437 1318 raft_consensus.cc:348] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.890552 1318 raft_consensus.cc:374] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.890585 1318 raft_consensus.cc:729] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.890705 1318 consensus_queue.cc:260] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.890957 1318 ts_tablet_manager.cc:1398] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:58:49.891798 1323 tablet_bootstrap.cc:492] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:49.891899 1323 ts_tablet_manager.cc:1367] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:49.892390 1323 raft_consensus.cc:348] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.892517 1323 raft_consensus.cc:374] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.892563 1323 raft_consensus.cc:729] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.892694 1323 consensus_queue.cc:260] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.892956 1323 ts_tablet_manager.cc:1398] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.893942 1318 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:49.894747 1317 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:49.894856 1317 ts_tablet_manager.cc:1367] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 01:58:49.895046 1319 tablet_bootstrap.cc:492] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:49.895099 1318 tablet_bootstrap.cc:654] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.895138 1319 ts_tablet_manager.cc:1367] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:58:49.895467 1317 raft_consensus.cc:348] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.895622 1317 raft_consensus.cc:374] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.895658 1317 raft_consensus.cc:729] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.895785 1317 consensus_queue.cc:260] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.895874 1319 raft_consensus.cc:348] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.895983 1319 raft_consensus.cc:374] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.896020 1319 raft_consensus.cc:729] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.896036 1317 ts_tablet_manager.cc:1398] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.896140 1319 consensus_queue.cc:260] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.896212 1317 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:49.896389 1319 ts_tablet_manager.cc:1398] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:49.896559 1319 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:49.897166 1317 tablet_bootstrap.cc:654] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.897938 1319 tablet_bootstrap.cc:654] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.898718 1317 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:49.898839 1317 ts_tablet_manager.cc:1367] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:49.899142 1319 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:49.899211 1319 ts_tablet_manager.cc:1367] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:49.899329 1317 raft_consensus.cc:348] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.899446 1317 raft_consensus.cc:374] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.899487 1317 raft_consensus.cc:729] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.899643 1317 consensus_queue.cc:260] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.899678 1319 raft_consensus.cc:348] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.899789 1319 raft_consensus.cc:374] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.899822 1319 raft_consensus.cc:729] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.899856 1318 tablet_bootstrap.cc:492] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:49.899914 1317 ts_tablet_manager.cc:1398] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.899940 1318 ts_tablet_manager.cc:1367] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 01:58:49.899991 1319 consensus_queue.cc:260] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.900346 1319 ts_tablet_manager.cc:1398] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:49.900380 1318 raft_consensus.cc:348] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.900496 1318 raft_consensus.cc:374] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.900534 1318 raft_consensus.cc:729] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.900651 1318 consensus_queue.cc:260] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.900893 1318 ts_tablet_manager.cc:1398] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:49.932399 1320 raft_consensus.cc:482] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.932724 1320 raft_consensus.cc:504] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.934319 1320 leader_election.cc:288] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:49.939182 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d0b13b23c7421ea802353cbaad45a7" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:49.939172 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d0b13b23c7421ea802353cbaad45a7" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:49.939337 1109 raft_consensus.cc:2454] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:49.939335 847 raft_consensus.cc:2454] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:49.939707 517 leader_election.cc:302] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:49.939934 1320 raft_consensus.cc:2781] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.940003 1320 raft_consensus.cc:482] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.940044 1320 raft_consensus.cc:3037] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.945983 1328 raft_consensus.cc:482] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.946105 1328 raft_consensus.cc:504] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.946588 1328 leader_election.cc:288] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:49.949362 1320 raft_consensus.cc:504] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.949869 1320 leader_election.cc:288] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:49.950698 1341 raft_consensus.cc:482] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.950793 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2aaee1132c40638fae59ad5f985056" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:49.950826 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2aaee1132c40638fae59ad5f985056" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:49.950943 979 raft_consensus.cc:2454] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:49.950920 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d0b13b23c7421ea802353cbaad45a7" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:49.950997 1243 raft_consensus.cc:2454] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:49.951058 847 raft_consensus.cc:3037] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.951174 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d0b13b23c7421ea802353cbaad45a7" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:49.950809 1341 raft_consensus.cc:504] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.951299 1109 raft_consensus.cc:3037] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.951687 1341 leader_election.cc:288] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:49.952082 1329 raft_consensus.cc:482] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:49.952185 1329 raft_consensus.cc:504] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.952349 651 leader_election.cc:302] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.952548 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:49.952658 979 raft_consensus.cc:2454] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:49.952853 1328 raft_consensus.cc:2781] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.952906 1328 raft_consensus.cc:482] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.952941 1328 raft_consensus.cc:3037] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.953383 1329 leader_election.cc:288] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:49.954361 1177 leader_election.cc:302] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.954526 1341 raft_consensus.cc:2781] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.954579 1341 raft_consensus.cc:482] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.954627 1341 raft_consensus.cc:3037] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.956610 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:49.956827 715 raft_consensus.cc:2454] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:49.958765 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:49.959064 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:49.959175 715 raft_consensus.cc:2454] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:49.959476 913 leader_election.cc:302] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:49.959726 1329 raft_consensus.cc:2781] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:49.959781 1329 raft_consensus.cc:482] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:49.959806 1329 raft_consensus.cc:3037] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.986704 847 raft_consensus.cc:2454] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:49.986708 1329 raft_consensus.cc:504] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.986708 1341 raft_consensus.cc:504] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.987237 1329 leader_election.cc:288] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:49.987231 1341 leader_election.cc:288] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:49.987478 517 leader_election.cc:302] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:49.987493 1328 raft_consensus.cc:504] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.987744 1320 raft_consensus.cc:2781] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.987934 1328 leader_election.cc:288] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:49.987943 1320 raft_consensus.cc:686] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:49.988191 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:49.988310 714 raft_consensus.cc:3037] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.988402 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2aaee1132c40638fae59ad5f985056" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:49.988320 1320 consensus_queue.cc:237] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:49.988509 1242 raft_consensus.cc:3037] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.989475 1242 raft_consensus.cc:2454] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:49.989475 714 raft_consensus.cc:2454] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:49.989713 1243 raft_consensus.cc:2379] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f8d9686ff2d42428d81510426857a12 in current term 1: Already voted for candidate b22eb38cc62f46d58c4bcdfb38ae776d in this term.
I20240503 01:58:49.989826 651 leader_election.cc:302] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.990015 1328 raft_consensus.cc:2781] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.990088 1328 raft_consensus.cc:686] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:49.990199 1328 consensus_queue.cc:237] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:49.991417 1177 leader_election.cc:302] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:49.991568 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:49.991669 1341 raft_consensus.cc:2781] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.991730 715 raft_consensus.cc:2379] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f8d9686ff2d42428d81510426857a12 in current term 1: Already voted for candidate b22eb38cc62f46d58c4bcdfb38ae776d in this term.
I20240503 01:58:49.991744 1341 raft_consensus.cc:686] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 LEADER]: Becoming Leader. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Running, Role: LEADER
I20240503 01:58:49.991870 1341 consensus_queue.cc:237] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:49.992015 441 catalog_manager.cc:5484] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:49.992197 1109 raft_consensus.cc:2454] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:49.992023 438 catalog_manager.cc:5484] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:49.992738 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2aaee1132c40638fae59ad5f985056" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:49.992852 979 raft_consensus.cc:3037] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.993101 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:49.993252 978 raft_consensus.cc:2379] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b22eb38cc62f46d58c4bcdfb38ae776d in current term 1: Already voted for candidate 4f8d9686ff2d42428d81510426857a12 in this term.
I20240503 01:58:49.994046 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edace0c39397435c929e99df6f2a5f0b" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:49.994658 440 catalog_manager.cc:5484] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d reported cstate change: term changed from 0 to 1, leader changed from <none> to b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:49.994997 915 leader_election.cc:302] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12; no voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d
I20240503 01:58:49.995528 1329 raft_consensus.cc:2726] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 01:58:50.048400 979 raft_consensus.cc:2454] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
W20240503 01:58:50.130350 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.130786 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.134654 1427 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:58:50.243167 979 raft_consensus.cc:1261] T edace0c39397435c929e99df6f2a5f0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.243501 715 raft_consensus.cc:1261] T edace0c39397435c929e99df6f2a5f0b P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.243940 1243 raft_consensus.cc:1261] T 2b2aaee1132c40638fae59ad5f985056 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.244241 1341 consensus_queue.cc:1035] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.244556 1341 consensus_queue.cc:1035] T edace0c39397435c929e99df6f2a5f0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.244613 1345 consensus_queue.cc:1035] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.244736 979 raft_consensus.cc:1261] T 2b2aaee1132c40638fae59ad5f985056 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.245235 1345 consensus_queue.cc:1035] T 2b2aaee1132c40638fae59ad5f985056 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.245776 1109 raft_consensus.cc:1261] T 57d0b13b23c7421ea802353cbaad45a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.246165 847 raft_consensus.cc:1261] T 57d0b13b23c7421ea802353cbaad45a7 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.246778 1320 consensus_queue.cc:1035] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.253692 1422 consensus_queue.cc:1035] T 57d0b13b23c7421ea802353cbaad45a7 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 147.161 ms
time per row: 0.0735807 ms
I20240503 01:58:50.338718 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:50.345640 959 tablet_service.cc:1467] Processing CreateTablet for tablet cccfe866337246b7882b876903570d0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.345746 1091 tablet_service.cc:1467] Processing CreateTablet for tablet ecc2fc58a14d4379831e4a94189c2693 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.345818 827 tablet_service.cc:1467] Processing CreateTablet for tablet ecc2fc58a14d4379831e4a94189c2693 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.345997 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc2fc58a14d4379831e4a94189c2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.345999 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccfe866337246b7882b876903570d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.346132 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc2fc58a14d4379831e4a94189c2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.348176 1319 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:50.348212 1316 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:50.348446 1323 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:50.349185 1316 tablet_bootstrap.cc:654] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.349455 1323 tablet_bootstrap.cc:654] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.349609 1319 tablet_bootstrap.cc:654] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.350510 695 tablet_service.cc:1467] Processing CreateTablet for tablet cccfe866337246b7882b876903570d0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.350778 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccfe866337246b7882b876903570d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.350823 1323 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:50.350914 1323 ts_tablet_manager.cc:1367] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:50.351366 1323 raft_consensus.cc:348] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.351488 1323 raft_consensus.cc:374] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.351523 1323 raft_consensus.cc:729] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.351701 1323 consensus_queue.cc:260] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.352030 1323 ts_tablet_manager.cc:1398] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:50.352661 1317 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:50.352861 1316 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:50.352942 1316 ts_tablet_manager.cc:1367] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:50.352901 1223 tablet_service.cc:1467] Processing CreateTablet for tablet cccfe866337246b7882b876903570d0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.353183 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccfe866337246b7882b876903570d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.353426 1316 raft_consensus.cc:348] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.353529 1316 raft_consensus.cc:374] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.353560 1316 raft_consensus.cc:729] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.353628 1317 tablet_bootstrap.cc:654] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.353660 1316 consensus_queue.cc:260] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.354650 1316 ts_tablet_manager.cc:1398] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:50.355235 1318 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:50.356307 1319 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:50.356441 1319 ts_tablet_manager.cc:1367] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:58:50.356462 1318 tablet_bootstrap.cc:654] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.357000 1319 raft_consensus.cc:348] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.357129 1319 raft_consensus.cc:374] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.357173 1319 raft_consensus.cc:729] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.357290 562 tablet_service.cc:1467] Processing CreateTablet for tablet ecc2fc58a14d4379831e4a94189c2693 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.357322 1319 consensus_queue.cc:260] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.357575 1318 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:50.357584 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc2fc58a14d4379831e4a94189c2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.357645 1319 ts_tablet_manager.cc:1398] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:50.357678 1318 ts_tablet_manager.cc:1367] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:58:50.358115 1318 raft_consensus.cc:348] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.358227 1318 raft_consensus.cc:374] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.358261 1318 raft_consensus.cc:729] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.358425 1318 consensus_queue.cc:260] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.358712 1318 ts_tablet_manager.cc:1398] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:50.359396 1314 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:50.359624 563 tablet_service.cc:1467] Processing CreateTablet for tablet c22514cbdd5b48b890e702b7b32e8910 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.359836 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22514cbdd5b48b890e702b7b32e8910. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.360338 1314 tablet_bootstrap.cc:654] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.361346 1314 tablet_bootstrap.cc:492] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:50.361431 1314 ts_tablet_manager.cc:1367] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:50.361811 1314 raft_consensus.cc:348] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.361907 1314 raft_consensus.cc:374] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.361935 1314 raft_consensus.cc:729] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.362041 1314 consensus_queue.cc:260] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.362280 1314 ts_tablet_manager.cc:1398] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:50.362437 1314 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:50.364019 1314 tablet_bootstrap.cc:654] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.364097 827 tablet_service.cc:1467] Processing CreateTablet for tablet c22514cbdd5b48b890e702b7b32e8910 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.364111 1091 tablet_service.cc:1467] Processing CreateTablet for tablet c22514cbdd5b48b890e702b7b32e8910 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dad96a5950614e11b31976e5225411f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:50.364382 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22514cbdd5b48b890e702b7b32e8910. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.364413 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22514cbdd5b48b890e702b7b32e8910. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:50.366777 1323 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:50.366967 1316 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:50.367784 1323 tablet_bootstrap.cc:654] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.367894 1317 tablet_bootstrap.cc:492] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:50.367986 1317 ts_tablet_manager.cc:1367] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:58:50.368499 1317 raft_consensus.cc:348] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.368614 1317 raft_consensus.cc:374] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.368657 1317 raft_consensus.cc:729] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.368762 1316 tablet_bootstrap.cc:654] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:50.368798 1317 consensus_queue.cc:260] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.369076 1317 ts_tablet_manager.cc:1398] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:50.369684 1323 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:50.369771 1323 ts_tablet_manager.cc:1367] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:50.370196 1323 raft_consensus.cc:348] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.370303 1323 raft_consensus.cc:374] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.370339 1323 raft_consensus.cc:729] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.370456 1323 consensus_queue.cc:260] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.370719 1323 ts_tablet_manager.cc:1398] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:50.371222 1316 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:50.371308 1316 ts_tablet_manager.cc:1367] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:50.371760 1316 raft_consensus.cc:348] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.371881 1316 raft_consensus.cc:374] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.371907 1316 raft_consensus.cc:729] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.372014 1316 consensus_queue.cc:260] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.372258 1316 ts_tablet_manager.cc:1398] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:50.374562 1314 tablet_bootstrap.cc:492] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:50.374646 1314 ts_tablet_manager.cc:1367] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240503 01:58:50.375016 1314 raft_consensus.cc:348] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.375114 1314 raft_consensus.cc:374] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:50.375154 1314 raft_consensus.cc:729] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:50.375255 1314 consensus_queue.cc:260] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.375512 1314 ts_tablet_manager.cc:1398] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:50.435961 1333 raft_consensus.cc:482] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:50.436127 1333 raft_consensus.cc:504] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.436635 1333 leader_election.cc:288] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:50.437098 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc2fc58a14d4379831e4a94189c2693" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:50.437121 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc2fc58a14d4379831e4a94189c2693" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:50.437237 1109 raft_consensus.cc:2454] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:50.437279 582 raft_consensus.cc:2454] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:50.437585 781 leader_election.cc:302] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:50.437757 1333 raft_consensus.cc:2781] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:50.437815 1333 raft_consensus.cc:482] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:50.437870 1333 raft_consensus.cc:3037] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.438822 1333 raft_consensus.cc:504] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.439141 1333 leader_election.cc:288] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:50.439478 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc2fc58a14d4379831e4a94189c2693" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:50.439479 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc2fc58a14d4379831e4a94189c2693" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:50.439625 582 raft_consensus.cc:3037] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.439644 1109 raft_consensus.cc:3037] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.440385 582 raft_consensus.cc:2454] T ecc2fc58a14d4379831e4a94189c2693 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:50.440397 1109 raft_consensus.cc:2454] T ecc2fc58a14d4379831e4a94189c2693 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:50.440670 781 leader_election.cc:302] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:50.440814 1333 raft_consensus.cc:2781] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:50.440954 1333 raft_consensus.cc:686] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [term 1 LEADER]: Becoming Leader. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Running, Role: LEADER
I20240503 01:58:50.441097 1333 consensus_queue.cc:237] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:50.442668 438 catalog_manager.cc:5484] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf77ccf223a649f590c7c2a7b632768c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "cf77ccf223a649f590c7c2a7b632768c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:50.583210 1380 raft_consensus.cc:482] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:50.583364 1380 raft_consensus.cc:504] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.584510 1380 leader_election.cc:288] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:50.588624 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22514cbdd5b48b890e702b7b32e8910" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:50.588816 847 raft_consensus.cc:2454] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:50.588958 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22514cbdd5b48b890e702b7b32e8910" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:50.589107 582 raft_consensus.cc:2454] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:50.589244 1045 leader_election.cc:302] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:50.589495 1380 raft_consensus.cc:2781] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:50.589561 1380 raft_consensus.cc:482] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:50.589601 1380 raft_consensus.cc:3037] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.591017 1380 raft_consensus.cc:504] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.591363 1380 leader_election.cc:288] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:50.591753 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22514cbdd5b48b890e702b7b32e8910" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:50.591885 847 raft_consensus.cc:3037] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.592643 847 raft_consensus.cc:2454] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:50.592882 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22514cbdd5b48b890e702b7b32e8910" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:50.592919 1045 leader_election.cc:302] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:50.593006 582 raft_consensus.cc:3037] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.593080 1380 raft_consensus.cc:2781] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:50.593250 1380 raft_consensus.cc:686] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:50.593572 1380 consensus_queue.cc:237] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:50.593976 582 raft_consensus.cc:2454] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:50.596088 438 catalog_manager.cc:5484] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:50.681573 1341 raft_consensus.cc:482] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:50.681710 1341 raft_consensus.cc:504] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.682109 1341 leader_election.cc:288] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:50.682495 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccfe866337246b7882b876903570d0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:50.682561 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccfe866337246b7882b876903570d0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:50.682657 979 raft_consensus.cc:2454] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:50.682686 715 raft_consensus.cc:2454] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:50.682953 1177 leader_election.cc:302] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:50.683112 1341 raft_consensus.cc:2781] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:50.683168 1341 raft_consensus.cc:482] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:50.683220 1341 raft_consensus.cc:3037] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.684072 1341 raft_consensus.cc:504] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.684432 1341 leader_election.cc:288] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:50.684875 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccfe866337246b7882b876903570d0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:50.684875 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccfe866337246b7882b876903570d0b" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:50.684989 979 raft_consensus.cc:3037] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.684989 715 raft_consensus.cc:3037] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:50.685909 715 raft_consensus.cc:2454] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:50.685909 979 raft_consensus.cc:2454] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:50.686182 1177 leader_election.cc:302] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:50.686337 1341 raft_consensus.cc:2781] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:50.686429 1341 raft_consensus.cc:686] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 LEADER]: Becoming Leader. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Running, Role: LEADER
I20240503 01:58:50.686568 1341 consensus_queue.cc:237] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:50.688618 438 catalog_manager.cc:5484] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d reported cstate change: term changed from 0 to 1, leader changed from <none> to b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:50.845377 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.845716 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.849053 1483 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:58:50.879571 1333 consensus_queue.cc:1035] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.882745 1475 consensus_queue.cc:1035] T ecc2fc58a14d4379831e4a94189c2693 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.930950 715 raft_consensus.cc:1261] T cccfe866337246b7882b876903570d0b P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.931228 979 raft_consensus.cc:1261] T cccfe866337246b7882b876903570d0b P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.931651 1331 consensus_queue.cc:1035] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.931802 1341 consensus_queue.cc:1035] T cccfe866337246b7882b876903570d0b P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.932530 582 raft_consensus.cc:1261] T c22514cbdd5b48b890e702b7b32e8910 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.933526 847 raft_consensus.cc:1261] T c22514cbdd5b48b890e702b7b32e8910 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:50.933920 1380 consensus_queue.cc:1035] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:50.934101 1480 consensus_queue.cc:1035] T c22514cbdd5b48b890e702b7b32e8910 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.086 ms
time per row: 0.055543 ms
I20240503 01:58:51.005331 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:51.011085 695 tablet_service.cc:1467] Processing CreateTablet for tablet fcd37670627f480a8883d3c448304ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.011085 959 tablet_service.cc:1467] Processing CreateTablet for tablet fcd37670627f480a8883d3c448304ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.011410 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd37670627f480a8883d3c448304ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.011420 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd37670627f480a8883d3c448304ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.011914 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 559fbdb613924b4ab7740c0560d9fe8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.012140 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559fbdb613924b4ab7740c0560d9fe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.013864 1553 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:51.013939 563 tablet_service.cc:1467] Processing CreateTablet for tablet a191cae8ecdd46ce8f49eb11fc967449 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.014021 1223 tablet_service.cc:1467] Processing CreateTablet for tablet fcd37670627f480a8883d3c448304ff4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.014199 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cae8ecdd46ce8f49eb11fc967449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.014261 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd37670627f480a8883d3c448304ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.015093 827 tablet_service.cc:1467] Processing CreateTablet for tablet a191cae8ecdd46ce8f49eb11fc967449 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.015362 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cae8ecdd46ce8f49eb11fc967449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.015368 1091 tablet_service.cc:1467] Processing CreateTablet for tablet a191cae8ecdd46ce8f49eb11fc967449 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.015663 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cae8ecdd46ce8f49eb11fc967449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.016669 1554 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:51.017586 695 tablet_service.cc:1467] Processing CreateTablet for tablet 559fbdb613924b4ab7740c0560d9fe8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.017832 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559fbdb613924b4ab7740c0560d9fe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.017797 959 tablet_service.cc:1467] Processing CreateTablet for tablet 559fbdb613924b4ab7740c0560d9fe8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=e15e6e135d1d4046909fc7a8872c84d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.018046 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559fbdb613924b4ab7740c0560d9fe8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.018088 1555 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:51.018216 1556 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:51.019004 1555 tablet_bootstrap.cc:654] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.019016 1556 tablet_bootstrap.cc:654] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.020139 1556 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:51.020231 1556 ts_tablet_manager.cc:1367] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:51.020247 1554 tablet_bootstrap.cc:654] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.020607 1553 tablet_bootstrap.cc:654] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.020722 1556 raft_consensus.cc:348] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.020841 1556 raft_consensus.cc:374] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.020874 1556 raft_consensus.cc:729] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.021003 1551 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:51.020990 1556 consensus_queue.cc:260] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.021283 1556 ts_tablet_manager.cc:1398] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:51.021724 1552 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:51.021953 1553 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:51.022111 1553 ts_tablet_manager.cc:1367] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:58:51.022406 1551 tablet_bootstrap.cc:654] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.022684 1554 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:51.022655 1553 raft_consensus.cc:348] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.022764 1553 raft_consensus.cc:374] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.022838 1554 ts_tablet_manager.cc:1367] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:51.022845 1553 raft_consensus.cc:729] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.023048 1553 consensus_queue.cc:260] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.023509 1553 ts_tablet_manager.cc:1398] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:51.023546 1551 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:51.023653 1551 ts_tablet_manager.cc:1367] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:51.023805 1553 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:51.024044 1551 raft_consensus.cc:348] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.024140 1551 raft_consensus.cc:374] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.024174 1551 raft_consensus.cc:729] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.024303 1551 consensus_queue.cc:260] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.024577 1551 ts_tablet_manager.cc:1398] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:51.024494 1554 raft_consensus.cc:348] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.024631 1554 raft_consensus.cc:374] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.024672 1554 raft_consensus.cc:729] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.024749 1551 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:51.024866 1554 consensus_queue.cc:260] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.025072 1553 tablet_bootstrap.cc:654] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.025189 1554 ts_tablet_manager.cc:1398] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:51.025586 1551 tablet_bootstrap.cc:654] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.026325 1552 tablet_bootstrap.cc:654] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.026624 1555 tablet_bootstrap.cc:492] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:51.026736 1555 ts_tablet_manager.cc:1367] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:58:51.027261 1555 raft_consensus.cc:348] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.027374 1555 raft_consensus.cc:374] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.027410 1555 raft_consensus.cc:729] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.027515 1551 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:51.027591 1555 consensus_queue.cc:260] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.027725 1551 ts_tablet_manager.cc:1367] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:58:51.028095 1551 raft_consensus.cc:348] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.028199 1551 raft_consensus.cc:374] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.028230 1551 raft_consensus.cc:729] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.028345 1551 consensus_queue.cc:260] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.028592 1551 ts_tablet_manager.cc:1398] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:51.028743 1555 ts_tablet_manager.cc:1398] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:51.030129 1552 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:51.030371 1552 ts_tablet_manager.cc:1367] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 01:58:51.030521 1553 tablet_bootstrap.cc:492] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:51.030685 1553 ts_tablet_manager.cc:1367] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:51.031023 1552 raft_consensus.cc:348] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.031271 1553 raft_consensus.cc:348] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.031313 1552 raft_consensus.cc:374] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.031401 1553 raft_consensus.cc:374] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.031404 1552 raft_consensus.cc:729] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.031477 1553 raft_consensus.cc:729] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.031652 1552 consensus_queue.cc:260] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.031703 1553 consensus_queue.cc:260] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.032269 1553 ts_tablet_manager.cc:1398] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:51.033895 1552 ts_tablet_manager.cc:1398] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:58:51.034111 1552 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:51.035159 1552 tablet_bootstrap.cc:654] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.036486 1552 tablet_bootstrap.cc:492] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:51.036563 1552 ts_tablet_manager.cc:1367] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:58:51.036929 1552 raft_consensus.cc:348] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.037019 1552 raft_consensus.cc:374] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.037040 1552 raft_consensus.cc:729] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.037163 1552 consensus_queue.cc:260] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.037437 1552 ts_tablet_manager.cc:1398] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:51.067404 1525 raft_consensus.cc:482] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.067582 1525 raft_consensus.cc:504] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.068003 1525 leader_election.cc:288] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:51.068480 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcd37670627f480a8883d3c448304ff4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:51.068480 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcd37670627f480a8883d3c448304ff4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:51.068641 1243 raft_consensus.cc:2454] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:51.068641 715 raft_consensus.cc:2454] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:51.068980 913 leader_election.cc:302] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:51.069167 1525 raft_consensus.cc:2781] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.069247 1525 raft_consensus.cc:482] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.069295 1525 raft_consensus.cc:3037] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.070343 1525 raft_consensus.cc:504] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.070693 1525 leader_election.cc:288] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:51.071062 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcd37670627f480a8883d3c448304ff4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:51.071097 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcd37670627f480a8883d3c448304ff4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:51.071197 1243 raft_consensus.cc:3037] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.071203 715 raft_consensus.cc:3037] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.072219 715 raft_consensus.cc:2454] T fcd37670627f480a8883d3c448304ff4 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:51.072219 1243 raft_consensus.cc:2454] T fcd37670627f480a8883d3c448304ff4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:51.072535 915 leader_election.cc:302] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:51.072726 1525 raft_consensus.cc:2781] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.072947 1525 raft_consensus.cc:686] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Running, Role: LEADER
I20240503 01:58:51.073269 1525 consensus_queue.cc:237] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:51.075439 441 catalog_manager.cc:5484] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9686ff2d42428d81510426857a12 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "4f8d9686ff2d42428d81510426857a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:51.113780 1380 raft_consensus.cc:482] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.113921 1380 raft_consensus.cc:504] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.114342 1380 leader_election.cc:288] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:51.114831 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cae8ecdd46ce8f49eb11fc967449" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:51.114838 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cae8ecdd46ce8f49eb11fc967449" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:51.114982 847 raft_consensus.cc:2454] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:51.114982 583 raft_consensus.cc:2454] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:51.115312 1047 leader_election.cc:302] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:51.115468 1380 raft_consensus.cc:2781] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.115532 1380 raft_consensus.cc:482] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.115605 1380 raft_consensus.cc:3037] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.116472 1380 raft_consensus.cc:504] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.116811 1380 leader_election.cc:288] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:51.117123 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cae8ecdd46ce8f49eb11fc967449" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:51.117245 847 raft_consensus.cc:3037] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.117257 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cae8ecdd46ce8f49eb11fc967449" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:51.117388 583 raft_consensus.cc:3037] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.118134 847 raft_consensus.cc:2454] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:51.118366 583 raft_consensus.cc:2454] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:51.118402 1045 leader_election.cc:302] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:51.118657 1380 raft_consensus.cc:2781] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.118760 1380 raft_consensus.cc:686] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:51.118899 1380 consensus_queue.cc:237] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:51.120584 441 catalog_manager.cc:5484] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:51.262044 1525 raft_consensus.cc:482] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.262223 1525 raft_consensus.cc:504] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.262645 1525 leader_election.cc:288] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:51.263120 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559fbdb613924b4ab7740c0560d9fe8f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:51.263120 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559fbdb613924b4ab7740c0560d9fe8f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:51.263254 715 raft_consensus.cc:2454] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:51.263271 1243 raft_consensus.cc:2454] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:51.263572 913 leader_election.cc:302] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:51.263810 1525 raft_consensus.cc:2781] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.263859 1525 raft_consensus.cc:482] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.263886 1525 raft_consensus.cc:3037] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.264748 1525 raft_consensus.cc:504] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.265095 1525 leader_election.cc:288] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:51.265442 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559fbdb613924b4ab7740c0560d9fe8f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:51.265440 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559fbdb613924b4ab7740c0560d9fe8f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:51.265560 715 raft_consensus.cc:3037] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.265559 1243 raft_consensus.cc:3037] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.266323 715 raft_consensus.cc:2454] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:51.266330 1243 raft_consensus.cc:2454] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:51.266602 915 leader_election.cc:302] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:51.266817 1525 raft_consensus.cc:2781] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.266892 1525 raft_consensus.cc:686] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Running, Role: LEADER
I20240503 01:58:51.267045 1525 consensus_queue.cc:237] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:51.268934 441 catalog_manager.cc:5484] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9686ff2d42428d81510426857a12 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "4f8d9686ff2d42428d81510426857a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:51.421085 1570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:51.421371 1570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:51.424701 1570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:58:51.493772 1525 consensus_queue.cc:1035] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.497828 1525 consensus_queue.cc:1035] T fcd37670627f480a8883d3c448304ff4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.508461 1242 raft_consensus.cc:1261] T 559fbdb613924b4ab7740c0560d9fe8f P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:51.508966 1566 consensus_queue.cc:1035] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.509518 714 raft_consensus.cc:1261] T 559fbdb613924b4ab7740c0560d9fe8f P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:51.514168 1566 consensus_queue.cc:1035] T 559fbdb613924b4ab7740c0560d9fe8f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.540053 847 raft_consensus.cc:1261] T a191cae8ecdd46ce8f49eb11fc967449 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:51.540053 583 raft_consensus.cc:1261] T a191cae8ecdd46ce8f49eb11fc967449 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:51.540532 1480 consensus_queue.cc:1035] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:51.540721 1380 consensus_queue.cc:1035] T a191cae8ecdd46ce8f49eb11fc967449 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.115 ms
time per row: 0.0540575 ms
I20240503 01:58:51.580137 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:51.586272 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 50b248b22783495a856627db92d92550 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.586272 827 tablet_service.cc:1467] Processing CreateTablet for tablet 50b248b22783495a856627db92d92550 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.586593 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b248b22783495a856627db92d92550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.586599 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b248b22783495a856627db92d92550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.586983 1223 tablet_service.cc:1467] Processing CreateTablet for tablet bb2933a9f17641a7a835a348e153da13 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.587246 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2933a9f17641a7a835a348e153da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.589243 563 tablet_service.cc:1467] Processing CreateTablet for tablet 50b248b22783495a856627db92d92550 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.589535 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b248b22783495a856627db92d92550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.589800 1637 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:51.590227 959 tablet_service.cc:1467] Processing CreateTablet for tablet bb2933a9f17641a7a835a348e153da13 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.590512 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2933a9f17641a7a835a348e153da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.590821 1637 tablet_bootstrap.cc:654] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.591907 695 tablet_service.cc:1467] Processing CreateTablet for tablet bb2933a9f17641a7a835a348e153da13 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.591946 562 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a5bb25be642dda808fde15d59dec9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.592149 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2933a9f17641a7a835a348e153da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.592160 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a5bb25be642dda808fde15d59dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.592881 1641 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:51.593784 1641 tablet_bootstrap.cc:654] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.594401 827 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a5bb25be642dda808fde15d59dec9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.594652 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a5bb25be642dda808fde15d59dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.594662 1640 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:51.595330 1640 tablet_bootstrap.cc:654] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.595523 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a5bb25be642dda808fde15d59dec9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=458aa15fdd4840a9aacbba7da7d1be80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.595829 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a5bb25be642dda808fde15d59dec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.595829 1636 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:51.596448 1637 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:51.596544 1637 ts_tablet_manager.cc:1367] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:51.596851 1636 tablet_bootstrap.cc:654] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.597038 1637 raft_consensus.cc:348] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.597141 1637 raft_consensus.cc:374] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.597180 1637 raft_consensus.cc:729] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.597298 1637 consensus_queue.cc:260] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.597445 1641 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:51.597591 1641 ts_tablet_manager.cc:1367] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:51.597604 1637 ts_tablet_manager.cc:1398] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:51.598140 1641 raft_consensus.cc:348] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.598366 1640 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:51.598359 1641 raft_consensus.cc:374] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.598414 1641 raft_consensus.cc:729] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.598461 1640 ts_tablet_manager.cc:1367] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:58:51.598549 1641 consensus_queue.cc:260] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.598835 1641 ts_tablet_manager.cc:1398] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:51.598906 1640 raft_consensus.cc:348] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.599018 1640 raft_consensus.cc:374] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.599048 1640 raft_consensus.cc:729] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.599159 1640 consensus_queue.cc:260] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.599280 1637 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:51.599426 1640 ts_tablet_manager.cc:1398] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:51.600225 1637 tablet_bootstrap.cc:654] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.600953 1638 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:51.601555 1637 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:51.601639 1637 ts_tablet_manager.cc:1367] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:51.601908 1638 tablet_bootstrap.cc:654] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.602119 1637 raft_consensus.cc:348] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.602229 1637 raft_consensus.cc:374] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.602263 1637 raft_consensus.cc:729] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.602385 1637 consensus_queue.cc:260] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.602622 1642 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:51.602900 1637 ts_tablet_manager.cc:1398] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:51.603431 1638 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:51.603611 1642 tablet_bootstrap.cc:654] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.603519 1638 ts_tablet_manager.cc:1367] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:51.604526 1640 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:51.604730 1642 tablet_bootstrap.cc:492] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:51.604806 1642 ts_tablet_manager.cc:1367] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:58:51.605226 1642 raft_consensus.cc:348] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.605337 1642 raft_consensus.cc:374] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.605374 1642 raft_consensus.cc:729] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.605494 1642 consensus_queue.cc:260] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.605782 1642 ts_tablet_manager.cc:1398] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:51.605953 1640 tablet_bootstrap.cc:654] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.607517 1636 tablet_bootstrap.cc:492] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:51.607662 1636 ts_tablet_manager.cc:1367] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:58:51.608094 1636 raft_consensus.cc:348] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.608194 1636 raft_consensus.cc:374] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.608224 1636 raft_consensus.cc:729] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.608352 1636 consensus_queue.cc:260] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.608664 1638 raft_consensus.cc:348] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.608850 1638 raft_consensus.cc:374] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.608924 1638 raft_consensus.cc:729] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.609148 1638 consensus_queue.cc:260] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.609444 1636 ts_tablet_manager.cc:1398] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:51.609665 1636 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:51.611310 1636 tablet_bootstrap.cc:654] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.611835 1638 ts_tablet_manager.cc:1398] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:58:51.612655 1640 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:51.612766 1640 ts_tablet_manager.cc:1367] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:58:51.613318 1636 tablet_bootstrap.cc:492] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:51.613281 1640 raft_consensus.cc:348] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.613399 1636 ts_tablet_manager.cc:1367] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:51.613399 1640 raft_consensus.cc:374] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.613438 1640 raft_consensus.cc:729] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.613582 1640 consensus_queue.cc:260] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.613852 1640 ts_tablet_manager.cc:1398] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:51.613857 1636 raft_consensus.cc:348] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.613989 1636 raft_consensus.cc:374] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.614040 1636 raft_consensus.cc:729] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.614163 1636 consensus_queue.cc:260] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.614439 1636 ts_tablet_manager.cc:1398] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:51.742025 1616 raft_consensus.cc:482] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.742189 1616 raft_consensus.cc:504] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.742660 1616 leader_election.cc:288] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:51.743139 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2933a9f17641a7a835a348e153da13" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:51.743141 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2933a9f17641a7a835a348e153da13" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:51.743295 1242 raft_consensus.cc:2454] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:51.743335 978 raft_consensus.cc:2454] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:51.743654 651 leader_election.cc:302] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:51.743841 1616 raft_consensus.cc:2781] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.743899 1616 raft_consensus.cc:482] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.743952 1616 raft_consensus.cc:3037] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.744834 1616 raft_consensus.cc:504] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.745164 1616 leader_election.cc:288] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:51.745595 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2933a9f17641a7a835a348e153da13" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:51.745607 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2933a9f17641a7a835a348e153da13" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:51.745723 1242 raft_consensus.cc:3037] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.745731 978 raft_consensus.cc:3037] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.746495 978 raft_consensus.cc:2454] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:51.746495 1242 raft_consensus.cc:2454] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:51.746800 651 leader_election.cc:302] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:51.746990 1616 raft_consensus.cc:2781] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.747076 1616 raft_consensus.cc:686] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:51.747228 1616 consensus_queue.cc:237] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:51.749055 441 catalog_manager.cc:5484] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:51.801453 1632 raft_consensus.cc:482] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.801594 1632 raft_consensus.cc:504] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.801999 1632 leader_election.cc:288] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:51.802461 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a5bb25be642dda808fde15d59dec9" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:51.802464 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a5bb25be642dda808fde15d59dec9" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:51.802614 847 raft_consensus.cc:2454] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:51.802634 583 raft_consensus.cc:2454] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:51.802971 1047 leader_election.cc:302] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:51.803138 1632 raft_consensus.cc:2781] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.803192 1632 raft_consensus.cc:482] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.803216 1632 raft_consensus.cc:3037] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.804093 1632 raft_consensus.cc:504] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.804421 1632 leader_election.cc:288] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:51.804757 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a5bb25be642dda808fde15d59dec9" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:51.804769 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a5bb25be642dda808fde15d59dec9" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:51.804869 847 raft_consensus.cc:3037] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.804880 583 raft_consensus.cc:3037] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.805589 847 raft_consensus.cc:2454] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:51.805680 583 raft_consensus.cc:2454] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:51.805856 1045 leader_election.cc:302] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:51.806063 1632 raft_consensus.cc:2781] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.806133 1632 raft_consensus.cc:686] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:51.806237 1632 consensus_queue.cc:237] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:51.807971 441 catalog_manager.cc:5484] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:51.922854 1511 raft_consensus.cc:482] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:51.922989 1511 raft_consensus.cc:504] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.923445 1511 leader_election.cc:288] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:51.924000 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b248b22783495a856627db92d92550" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:51.924019 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b248b22783495a856627db92d92550" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:51.924168 847 raft_consensus.cc:2454] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:51.924168 1109 raft_consensus.cc:2454] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:51.924501 517 leader_election.cc:302] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:51.924665 1511 raft_consensus.cc:2781] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:51.924721 1511 raft_consensus.cc:482] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:51.924770 1511 raft_consensus.cc:3037] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.925626 1511 raft_consensus.cc:504] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.925940 1511 leader_election.cc:288] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:51.926338 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b248b22783495a856627db92d92550" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:51.926338 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b248b22783495a856627db92d92550" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:51.926465 1109 raft_consensus.cc:3037] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.926465 847 raft_consensus.cc:3037] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.927413 847 raft_consensus.cc:2454] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:51.927413 1109 raft_consensus.cc:2454] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:51.927712 517 leader_election.cc:302] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:51.927865 1511 raft_consensus.cc:2781] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.927953 1511 raft_consensus.cc:686] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:51.928093 1511 consensus_queue.cc:237] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:51.929962 441 catalog_manager.cc:5484] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:52.079753 1655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:52.080065 1655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:52.083388 1655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:58:52.176715 1109 raft_consensus.cc:1261] T 50b248b22783495a856627db92d92550 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.176748 847 raft_consensus.cc:1261] T 50b248b22783495a856627db92d92550 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.177446 1511 consensus_queue.cc:1035] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.177716 1646 consensus_queue.cc:1035] T 50b248b22783495a856627db92d92550 P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.181072 846 raft_consensus.cc:1261] T 0f0a5bb25be642dda808fde15d59dec9 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.181834 1632 consensus_queue.cc:1035] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.181926 583 raft_consensus.cc:1261] T 0f0a5bb25be642dda808fde15d59dec9 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.182359 1632 consensus_queue.cc:1035] T 0f0a5bb25be642dda808fde15d59dec9 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.188228 1242 raft_consensus.cc:1261] T bb2933a9f17641a7a835a348e153da13 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.189332 1463 consensus_queue.cc:1035] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.190757 978 raft_consensus.cc:1261] T bb2933a9f17641a7a835a348e153da13 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.191224 1616 consensus_queue.cc:1035] T bb2933a9f17641a7a835a348e153da13 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.154 ms
time per row: 0.065577 ms
I20240503 01:58:52.260231 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:52.266538 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 40de9f5424f04345a31ddbcff2f5f45c (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.266916 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40de9f5424f04345a31ddbcff2f5f45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.266822 959 tablet_service.cc:1467] Processing CreateTablet for tablet 40de9f5424f04345a31ddbcff2f5f45c (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.267148 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40de9f5424f04345a31ddbcff2f5f45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.267783 1222 tablet_service.cc:1467] Processing CreateTablet for tablet f504aaf5826842988eacf0e4d13de3f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.267760 694 tablet_service.cc:1467] Processing CreateTablet for tablet f504aaf5826842988eacf0e4d13de3f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.267822 695 tablet_service.cc:1467] Processing CreateTablet for tablet 40de9f5424f04345a31ddbcff2f5f45c (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.268003 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f504aaf5826842988eacf0e4d13de3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.268026 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f504aaf5826842988eacf0e4d13de3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.268390 562 tablet_service.cc:1467] Processing CreateTablet for tablet 04a6b21333ec49eca8e66bfadec8f57a (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.268678 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a6b21333ec49eca8e66bfadec8f57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.269816 958 tablet_service.cc:1467] Processing CreateTablet for tablet f504aaf5826842988eacf0e4d13de3f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.270048 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f504aaf5826842988eacf0e4d13de3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.270323 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 04a6b21333ec49eca8e66bfadec8f57a (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.270473 827 tablet_service.cc:1467] Processing CreateTablet for tablet 04a6b21333ec49eca8e66bfadec8f57a (DEFAULT_TABLE table=rebalance_test_table_5 [id=152428b8017c48fb933a8643f560e0c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.270586 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a6b21333ec49eca8e66bfadec8f57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.270733 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a6b21333ec49eca8e66bfadec8f57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.271296 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40de9f5424f04345a31ddbcff2f5f45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.272397 1718 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:52.273558 1718 tablet_bootstrap.cc:654] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.273718 1721 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:52.274070 1720 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:52.274336 1719 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:52.274821 1721 tablet_bootstrap.cc:654] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.275039 1720 tablet_bootstrap.cc:654] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.275312 1719 tablet_bootstrap.cc:654] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.276057 1724 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:52.276335 1720 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:52.276432 1720 ts_tablet_manager.cc:1367] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:52.276655 1717 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:52.276895 1720 raft_consensus.cc:348] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.276988 1720 raft_consensus.cc:374] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.277012 1720 raft_consensus.cc:729] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.277050 1724 tablet_bootstrap.cc:654] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.277138 1720 consensus_queue.cc:260] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.277447 1720 ts_tablet_manager.cc:1398] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:52.277539 1717 tablet_bootstrap.cc:654] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.278115 1718 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:52.278244 1718 ts_tablet_manager.cc:1367] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:52.278740 1718 raft_consensus.cc:348] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.278929 1718 raft_consensus.cc:374] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.279014 1719 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:52.279021 1718 raft_consensus.cc:729] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.279202 1719 ts_tablet_manager.cc:1367] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:52.279206 1718 consensus_queue.cc:260] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.279453 1724 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:52.279605 1724 ts_tablet_manager.cc:1367] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:52.279812 1719 raft_consensus.cc:348] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.280032 1719 raft_consensus.cc:374] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.280123 1719 raft_consensus.cc:729] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.280102 1724 raft_consensus.cc:348] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.280347 1719 consensus_queue.cc:260] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.280570 1724 raft_consensus.cc:374] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.281067 1717 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:52.281067 1718 ts_tablet_manager.cc:1398] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:52.281239 1718 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:52.281877 1719 ts_tablet_manager.cc:1398] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:52.282119 1719 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:52.282263 1718 tablet_bootstrap.cc:654] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.282392 1717 ts_tablet_manager.cc:1367] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:52.283560 1721 tablet_bootstrap.cc:492] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:52.283661 1721 ts_tablet_manager.cc:1367] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:58:52.283795 1719 tablet_bootstrap.cc:654] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.283849 1717 raft_consensus.cc:348] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.283962 1717 raft_consensus.cc:374] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.283993 1717 raft_consensus.cc:729] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.284116 1717 consensus_queue.cc:260] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.284121 1721 raft_consensus.cc:348] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.284210 1721 raft_consensus.cc:374] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.284240 1721 raft_consensus.cc:729] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.284400 1721 consensus_queue.cc:260] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.284490 1717 ts_tablet_manager.cc:1398] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:52.284653 1717 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:52.284657 1721 ts_tablet_manager.cc:1398] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.284996 1719 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:52.285077 1719 ts_tablet_manager.cc:1367] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:52.285573 1719 raft_consensus.cc:348] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.285681 1719 raft_consensus.cc:374] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.285715 1719 raft_consensus.cc:729] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.285828 1719 consensus_queue.cc:260] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.285996 1718 tablet_bootstrap.cc:492] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:52.286038 1717 tablet_bootstrap.cc:654] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.286075 1718 ts_tablet_manager.cc:1367] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:52.286123 1719 ts_tablet_manager.cc:1398] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.286629 1724 raft_consensus.cc:729] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.286644 1718 raft_consensus.cc:348] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.286742 1718 raft_consensus.cc:374] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.286820 1718 raft_consensus.cc:729] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.286861 1724 consensus_queue.cc:260] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.286971 1718 consensus_queue.cc:260] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.287906 1717 tablet_bootstrap.cc:492] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:52.288053 1717 ts_tablet_manager.cc:1367] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:52.288511 1717 raft_consensus.cc:348] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.288674 1718 ts_tablet_manager.cc:1398] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:52.288841 1717 raft_consensus.cc:374] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.288889 1717 raft_consensus.cc:729] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.289136 1717 consensus_queue.cc:260] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.289546 1717 ts_tablet_manager.cc:1398] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.289834 1724 ts_tablet_manager.cc:1398] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 01:58:52.295425 1511 raft_consensus.cc:482] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:52.295585 1511 raft_consensus.cc:504] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.295949 1511 leader_election.cc:288] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:52.296269 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a6b21333ec49eca8e66bfadec8f57a" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:52.296389 847 raft_consensus.cc:2454] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:52.296360 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a6b21333ec49eca8e66bfadec8f57a" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:52.296517 1109 raft_consensus.cc:2454] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:52.296658 517 leader_election.cc:302] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:52.296821 1511 raft_consensus.cc:2781] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:52.296880 1511 raft_consensus.cc:482] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:52.296909 1511 raft_consensus.cc:3037] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.297789 1511 raft_consensus.cc:504] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.298115 1511 leader_election.cc:288] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:52.298564 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a6b21333ec49eca8e66bfadec8f57a" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:52.298560 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a6b21333ec49eca8e66bfadec8f57a" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:52.298681 1109 raft_consensus.cc:3037] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.298681 847 raft_consensus.cc:3037] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.299640 847 raft_consensus.cc:2454] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:52.299640 1109 raft_consensus.cc:2454] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:52.299896 517 leader_election.cc:302] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:52.300055 1511 raft_consensus.cc:2781] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.300172 1511 raft_consensus.cc:686] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:52.300325 1511 consensus_queue.cc:237] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } }
I20240503 01:58:52.301972 436 catalog_manager.cc:5484] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:52.340634 1596 raft_consensus.cc:482] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:52.340768 1596 raft_consensus.cc:504] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.341145 1596 leader_election.cc:288] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:52.341565 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40de9f5424f04345a31ddbcff2f5f45c" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:52.341638 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40de9f5424f04345a31ddbcff2f5f45c" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:52.341702 978 raft_consensus.cc:2454] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:52.341794 715 raft_consensus.cc:2454] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:52.342012 1177 leader_election.cc:302] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.342259 1596 raft_consensus.cc:2781] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:52.342332 1596 raft_consensus.cc:482] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:52.342386 1596 raft_consensus.cc:3037] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.343245 1596 raft_consensus.cc:504] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.343535 1596 leader_election.cc:288] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:52.343883 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40de9f5424f04345a31ddbcff2f5f45c" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:52.343897 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40de9f5424f04345a31ddbcff2f5f45c" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:52.343988 715 raft_consensus.cc:3037] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.343997 978 raft_consensus.cc:3037] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.344717 715 raft_consensus.cc:2454] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:52.344930 978 raft_consensus.cc:2454] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:52.344996 1177 leader_election.cc:302] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.345139 1596 raft_consensus.cc:2781] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.345384 1596 raft_consensus.cc:686] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 LEADER]: Becoming Leader. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Running, Role: LEADER
I20240503 01:58:52.345551 1596 consensus_queue.cc:237] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.347232 436 catalog_manager.cc:5484] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d reported cstate change: term changed from 0 to 1, leader changed from <none> to b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:52.464702 1616 raft_consensus.cc:482] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:52.464826 1616 raft_consensus.cc:504] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.465365 1616 leader_election.cc:288] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:52.465802 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f504aaf5826842988eacf0e4d13de3f6" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:52.465802 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f504aaf5826842988eacf0e4d13de3f6" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:52.465953 978 raft_consensus.cc:2454] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:52.465953 1242 raft_consensus.cc:2454] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:52.466252 651 leader_election.cc:302] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.466420 1616 raft_consensus.cc:2781] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:52.466495 1616 raft_consensus.cc:482] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:52.466526 1616 raft_consensus.cc:3037] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.467383 1616 raft_consensus.cc:504] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.467751 1616 leader_election.cc:288] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:52.468124 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f504aaf5826842988eacf0e4d13de3f6" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:52.468124 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f504aaf5826842988eacf0e4d13de3f6" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:52.468257 1242 raft_consensus.cc:3037] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.468322 978 raft_consensus.cc:3037] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.469220 1242 raft_consensus.cc:2454] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:52.469220 978 raft_consensus.cc:2454] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:52.469556 651 leader_election.cc:302] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.469779 1616 raft_consensus.cc:2781] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.469849 1616 raft_consensus.cc:686] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:52.470002 1616 consensus_queue.cc:237] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:52.471900 436 catalog_manager.cc:5484] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:52.622459 1736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:52.622742 1736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:52.625936 1736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 01:58:52.707608 715 raft_consensus.cc:1261] T 40de9f5424f04345a31ddbcff2f5f45c P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.708007 1242 raft_consensus.cc:1261] T f504aaf5826842988eacf0e4d13de3f6 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.708182 979 raft_consensus.cc:1261] T 40de9f5424f04345a31ddbcff2f5f45c P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer b22eb38cc62f46d58c4bcdfb38ae776d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.708343 978 raft_consensus.cc:1261] T f504aaf5826842988eacf0e4d13de3f6 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.708824 1616 consensus_queue.cc:1035] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.709003 1463 consensus_queue.cc:1035] T f504aaf5826842988eacf0e4d13de3f6 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.712384 1774 consensus_queue.cc:1035] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.712867 1733 consensus_queue.cc:1035] T 40de9f5424f04345a31ddbcff2f5f45c P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.715014 1109 raft_consensus.cc:1261] T 04a6b21333ec49eca8e66bfadec8f57a P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.715478 847 raft_consensus.cc:1261] T 04a6b21333ec49eca8e66bfadec8f57a P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:52.716593 1646 consensus_queue.cc:1035] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:52.716729 1511 consensus_queue.cc:1035] T 04a6b21333ec49eca8e66bfadec8f57a P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.083 ms
time per row: 0.0540413 ms
I20240503 01:58:52.780519 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:52.786569 827 tablet_service.cc:1467] Processing CreateTablet for tablet bd3010b22936401d8d368e758e93154e (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.786569 1091 tablet_service.cc:1467] Processing CreateTablet for tablet bd3010b22936401d8d368e758e93154e (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.786569 562 tablet_service.cc:1467] Processing CreateTablet for tablet bd3010b22936401d8d368e758e93154e (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.786901 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3010b22936401d8d368e758e93154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.786901 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3010b22936401d8d368e758e93154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.786906 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3010b22936401d8d368e758e93154e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.788910 1789 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:52.789191 1724 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:52.789742 1223 tablet_service.cc:1467] Processing CreateTablet for tablet ba7691f939444a5ea03af2c66d4dc6ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.789969 1789 tablet_bootstrap.cc:654] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.789988 1724 tablet_bootstrap.cc:654] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.790021 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7691f939444a5ea03af2c66d4dc6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.790611 695 tablet_service.cc:1467] Processing CreateTablet for tablet ba7691f939444a5ea03af2c66d4dc6ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.790611 958 tablet_service.cc:1467] Processing CreateTablet for tablet ba7691f939444a5ea03af2c66d4dc6ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.790859 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7691f939444a5ea03af2c66d4dc6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.790859 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7691f939444a5ea03af2c66d4dc6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.791558 827 tablet_service.cc:1467] Processing CreateTablet for tablet 7905b8572e194a87bb263f629b311f93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.791779 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905b8572e194a87bb263f629b311f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.792990 1793 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:52.793928 1793 tablet_bootstrap.cc:654] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.793886 562 tablet_service.cc:1467] Processing CreateTablet for tablet 7905b8572e194a87bb263f629b311f93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.794142 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905b8572e194a87bb263f629b311f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.794288 1789 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:52.794392 1789 ts_tablet_manager.cc:1367] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:58:52.794770 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 7905b8572e194a87bb263f629b311f93 (DEFAULT_TABLE table=rebalance_test_table_6 [id=753105b44e82443b8ceaba1e8d80d2d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:52.794813 1789 raft_consensus.cc:348] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.794907 1789 raft_consensus.cc:374] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.794948 1789 raft_consensus.cc:729] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.795014 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7905b8572e194a87bb263f629b311f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:52.795082 1789 consensus_queue.cc:260] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.795387 1789 ts_tablet_manager.cc:1398] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.796216 1793 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:52.796342 1793 ts_tablet_manager.cc:1367] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:52.797379 1793 raft_consensus.cc:348] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.797514 1789 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:52.797529 1793 raft_consensus.cc:374] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.797603 1793 raft_consensus.cc:729] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.797684 1794 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:52.797796 1793 consensus_queue.cc:260] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.798475 1789 tablet_bootstrap.cc:654] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.798696 1793 ts_tablet_manager.cc:1398] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:52.799146 1724 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:52.799240 1724 ts_tablet_manager.cc:1367] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 01:58:52.799612 1789 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:52.799683 1789 ts_tablet_manager.cc:1367] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:52.799729 1724 raft_consensus.cc:348] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.799832 1724 raft_consensus.cc:374] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.799861 1724 raft_consensus.cc:729] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.799980 1724 consensus_queue.cc:260] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.800089 1789 raft_consensus.cc:348] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.800170 1789 raft_consensus.cc:374] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.800211 1789 raft_consensus.cc:729] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.800266 1724 ts_tablet_manager.cc:1398] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.800323 1789 consensus_queue.cc:260] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.800429 1724 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:52.800578 1789 ts_tablet_manager.cc:1398] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.801296 1794 tablet_bootstrap.cc:654] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.801338 1724 tablet_bootstrap.cc:654] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.801944 1792 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:52.802743 1792 tablet_bootstrap.cc:654] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.802870 1724 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:52.802968 1724 ts_tablet_manager.cc:1367] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:52.803381 1724 raft_consensus.cc:348] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.803505 1724 raft_consensus.cc:374] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.803580 1724 raft_consensus.cc:729] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.803694 1794 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:52.803696 1724 consensus_queue.cc:260] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.803781 1794 ts_tablet_manager.cc:1367] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:52.803956 1724 ts_tablet_manager.cc:1398] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.804217 1794 raft_consensus.cc:348] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.804329 1794 raft_consensus.cc:374] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.804360 1794 raft_consensus.cc:729] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.804486 1794 consensus_queue.cc:260] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.805231 1794 ts_tablet_manager.cc:1398] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:52.805466 1792 tablet_bootstrap.cc:492] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:52.805510 1790 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:52.805565 1792 ts_tablet_manager.cc:1367] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:52.806217 1792 raft_consensus.cc:348] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.806339 1792 raft_consensus.cc:374] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.806404 1792 raft_consensus.cc:729] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.806562 1792 consensus_queue.cc:260] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.806818 1792 ts_tablet_manager.cc:1398] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:52.807643 1790 tablet_bootstrap.cc:654] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.808807 1790 tablet_bootstrap.cc:492] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:52.808905 1790 ts_tablet_manager.cc:1367] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:58:52.809315 1790 raft_consensus.cc:348] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.809440 1790 raft_consensus.cc:374] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.809473 1790 raft_consensus.cc:729] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.809616 1790 consensus_queue.cc:260] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.809949 1790 ts_tablet_manager.cc:1398] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:52.810151 1790 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:52.811043 1790 tablet_bootstrap.cc:654] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:52.812476 1790 tablet_bootstrap.cc:492] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:52.812548 1790 ts_tablet_manager.cc:1367] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:58:52.812884 1790 raft_consensus.cc:348] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.813010 1790 raft_consensus.cc:374] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:52.813059 1790 raft_consensus.cc:729] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:52.813194 1790 consensus_queue.cc:260] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.813447 1790 ts_tablet_manager.cc:1398] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:52.879307 1774 raft_consensus.cc:482] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:52.879696 1774 raft_consensus.cc:504] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.880623 1774 leader_election.cc:288] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:52.881300 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7691f939444a5ea03af2c66d4dc6ef" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:52.881371 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7691f939444a5ea03af2c66d4dc6ef" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:52.881440 978 raft_consensus.cc:2454] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:52.881518 715 raft_consensus.cc:2454] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 0.
I20240503 01:58:52.881767 1177 leader_election.cc:302] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.881961 1774 raft_consensus.cc:2781] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:52.882018 1774 raft_consensus.cc:482] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:52.882071 1774 raft_consensus.cc:3037] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.882917 1774 raft_consensus.cc:504] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.883230 1774 leader_election.cc:288] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979)
I20240503 01:58:52.883579 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7691f939444a5ea03af2c66d4dc6ef" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:52.883615 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7691f939444a5ea03af2c66d4dc6ef" candidate_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:52.883697 978 raft_consensus.cc:3037] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.883711 715 raft_consensus.cc:3037] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.884418 978 raft_consensus.cc:2454] T ba7691f939444a5ea03af2c66d4dc6ef P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:52.884467 715 raft_consensus.cc:2454] T ba7691f939444a5ea03af2c66d4dc6ef P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22eb38cc62f46d58c4bcdfb38ae776d in term 1.
I20240503 01:58:52.884682 1177 leader_election.cc:302] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:52.884840 1774 raft_consensus.cc:2781] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.884927 1774 raft_consensus.cc:686] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 LEADER]: Becoming Leader. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Running, Role: LEADER
I20240503 01:58:52.885241 1774 consensus_queue.cc:237] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:52.887012 436 catalog_manager.cc:5484] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d reported cstate change: term changed from 0 to 1, leader changed from <none> to b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:52.942941 1532 raft_consensus.cc:482] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:52.943095 1532 raft_consensus.cc:504] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.943655 1532 leader_election.cc:288] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:52.944094 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3010b22936401d8d368e758e93154e" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:52.944096 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3010b22936401d8d368e758e93154e" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:52.944252 1109 raft_consensus.cc:2454] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:52.944258 583 raft_consensus.cc:2454] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:52.944630 781 leader_election.cc:302] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:52.944816 1532 raft_consensus.cc:2781] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:52.944875 1532 raft_consensus.cc:482] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:52.944903 1532 raft_consensus.cc:3037] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.945760 1532 raft_consensus.cc:504] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.946113 1532 leader_election.cc:288] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453)
I20240503 01:58:52.946573 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3010b22936401d8d368e758e93154e" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:52.946573 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3010b22936401d8d368e758e93154e" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:52.946710 1109 raft_consensus.cc:3037] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.946709 583 raft_consensus.cc:3037] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:52.947731 1109 raft_consensus.cc:2454] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:52.947731 583 raft_consensus.cc:2454] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:52.948074 781 leader_election.cc:302] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:52.948251 1532 raft_consensus.cc:2781] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:52.948424 1532 raft_consensus.cc:686] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [term 1 LEADER]: Becoming Leader. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Running, Role: LEADER
I20240503 01:58:52.948573 1532 consensus_queue.cc:237] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:52.950384 436 catalog_manager.cc:5484] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf77ccf223a649f590c7c2a7b632768c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "cf77ccf223a649f590c7c2a7b632768c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:53.124819 1532 raft_consensus.cc:482] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:53.124953 1532 raft_consensus.cc:504] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.125439 1532 leader_election.cc:288] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:53.125931 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905b8572e194a87bb263f629b311f93" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:53.125932 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905b8572e194a87bb263f629b311f93" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:53.126075 583 raft_consensus.cc:2454] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:53.126102 1109 raft_consensus.cc:2454] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 0.
I20240503 01:58:53.126415 783 leader_election.cc:302] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:53.126587 1532 raft_consensus.cc:2781] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:53.126652 1532 raft_consensus.cc:482] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:53.126700 1532 raft_consensus.cc:3037] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.127487 1532 raft_consensus.cc:504] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.127911 1532 leader_election.cc:288] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283)
I20240503 01:58:53.128304 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905b8572e194a87bb263f629b311f93" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:53.128312 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7905b8572e194a87bb263f629b311f93" candidate_uuid: "cf77ccf223a649f590c7c2a7b632768c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:53.128425 583 raft_consensus.cc:3037] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.128432 1109 raft_consensus.cc:3037] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.129335 1109 raft_consensus.cc:2454] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:53.129335 583 raft_consensus.cc:2454] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf77ccf223a649f590c7c2a7b632768c in term 1.
I20240503 01:58:53.129642 781 leader_election.cc:302] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:53.129886 1532 raft_consensus.cc:2781] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:53.129984 1532 raft_consensus.cc:686] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [term 1 LEADER]: Becoming Leader. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Running, Role: LEADER
I20240503 01:58:53.130151 1532 consensus_queue.cc:237] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.132202 436 catalog_manager.cc:5484] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf77ccf223a649f590c7c2a7b632768c (127.0.99.131). New cstate: current_term: 1 leader_uuid: "cf77ccf223a649f590c7c2a7b632768c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:53.279062 1807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.279335 1807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.282979 1807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 01:58:53.306850 1774 consensus_queue.cc:1035] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.310854 1596 consensus_queue.cc:1035] T ba7691f939444a5ea03af2c66d4dc6ef P b22eb38cc62f46d58c4bcdfb38ae776d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.376921 583 raft_consensus.cc:1261] T bd3010b22936401d8d368e758e93154e P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.377622 1109 raft_consensus.cc:1261] T bd3010b22936401d8d368e758e93154e P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.380206 1532 consensus_queue.cc:1035] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.380622 1532 consensus_queue.cc:1035] T bd3010b22936401d8d368e758e93154e P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.381278 583 raft_consensus.cc:1261] T 7905b8572e194a87bb263f629b311f93 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.382325 1804 consensus_queue.cc:1035] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.384828 1109 raft_consensus.cc:1261] T 7905b8572e194a87bb263f629b311f93 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer cf77ccf223a649f590c7c2a7b632768c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.386698 1804 consensus_queue.cc:1035] T 7905b8572e194a87bb263f629b311f93 P cf77ccf223a649f590c7c2a7b632768c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.213 ms
time per row: 0.0556065 ms
I20240503 01:58:53.445212 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:53.451017 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 00800c590d72473da73a658b54bf7d77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.451172 694 tablet_service.cc:1467] Processing CreateTablet for tablet e071aab00f4742629e47b7aa2d6ef730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.451393 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00800c590d72473da73a658b54bf7d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.451426 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071aab00f4742629e47b7aa2d6ef730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.451450 695 tablet_service.cc:1467] Processing CreateTablet for tablet 00800c590d72473da73a658b54bf7d77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.451694 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00800c590d72473da73a658b54bf7d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.453372 1870 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:53.454077 562 tablet_service.cc:1467] Processing CreateTablet for tablet 082691bb568745b382415f1380cf1383 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.454247 1870 tablet_bootstrap.cc:654] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.454416 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082691bb568745b382415f1380cf1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.454521 1871 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:53.455179 827 tablet_service.cc:1467] Processing CreateTablet for tablet 082691bb568745b382415f1380cf1383 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.455440 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082691bb568745b382415f1380cf1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.455492 1871 tablet_bootstrap.cc:654] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.456976 1874 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:53.457422 1871 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:53.457522 1871 ts_tablet_manager.cc:1367] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:53.457548 1875 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:53.458155 1871 raft_consensus.cc:348] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.458269 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 082691bb568745b382415f1380cf1383 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.458431 1871 raft_consensus.cc:374] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.458487 1874 tablet_bootstrap.cc:654] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.458536 1871 raft_consensus.cc:729] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.458559 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082691bb568745b382415f1380cf1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.458750 1871 consensus_queue.cc:260] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.458885 1875 tablet_bootstrap.cc:654] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.459226 1871 ts_tablet_manager.cc:1398] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:53.460322 1874 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:53.460419 1874 ts_tablet_manager.cc:1367] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:53.460774 1223 tablet_service.cc:1467] Processing CreateTablet for tablet e071aab00f4742629e47b7aa2d6ef730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.460855 1875 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:53.460934 1875 ts_tablet_manager.cc:1367] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:53.460873 1874 raft_consensus.cc:348] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.460978 1874 raft_consensus.cc:374] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.461009 1874 raft_consensus.cc:729] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.461019 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071aab00f4742629e47b7aa2d6ef730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.461138 1874 consensus_queue.cc:260] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.461237 958 tablet_service.cc:1467] Processing CreateTablet for tablet 00800c590d72473da73a658b54bf7d77 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.461407 1874 ts_tablet_manager.cc:1398] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:53.461357 1875 raft_consensus.cc:348] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.461472 1875 raft_consensus.cc:374] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.461510 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00800c590d72473da73a658b54bf7d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.461505 1875 raft_consensus.cc:729] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.461627 1875 consensus_queue.cc:260] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.461894 1875 ts_tablet_manager.cc:1398] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:53.463101 1871 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:53.463773 1870 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:53.463783 959 tablet_service.cc:1467] Processing CreateTablet for tablet e071aab00f4742629e47b7aa2d6ef730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a87143f2c8dd4ccb81cb3161b9195a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.463871 1870 ts_tablet_manager.cc:1367] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:58:53.463994 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071aab00f4742629e47b7aa2d6ef730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.464165 1871 tablet_bootstrap.cc:654] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.464346 1870 raft_consensus.cc:348] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.464464 1870 raft_consensus.cc:374] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.464504 1870 raft_consensus.cc:729] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.464669 1870 consensus_queue.cc:260] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.464977 1870 ts_tablet_manager.cc:1398] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:53.465150 1870 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:53.465343 1871 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:53.465413 1871 ts_tablet_manager.cc:1367] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:53.465842 1871 raft_consensus.cc:348] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.465935 1871 raft_consensus.cc:374] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.465975 1871 raft_consensus.cc:729] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.466120 1871 consensus_queue.cc:260] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.466331 1870 tablet_bootstrap.cc:654] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.466485 1871 ts_tablet_manager.cc:1398] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:53.467360 1870 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:53.467442 1870 ts_tablet_manager.cc:1367] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:53.467936 1870 raft_consensus.cc:348] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.468089 1870 raft_consensus.cc:374] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.468134 1879 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:53.468168 1870 raft_consensus.cc:729] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.468355 1870 consensus_queue.cc:260] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.468648 1870 ts_tablet_manager.cc:1398] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:53.469892 1879 tablet_bootstrap.cc:654] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.469120 1878 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:53.471274 1879 tablet_bootstrap.cc:492] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:53.471360 1879 ts_tablet_manager.cc:1367] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:53.471750 1878 tablet_bootstrap.cc:654] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.471803 1879 raft_consensus.cc:348] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.471940 1879 raft_consensus.cc:374] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.472004 1879 raft_consensus.cc:729] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.472139 1879 consensus_queue.cc:260] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.472483 1879 ts_tablet_manager.cc:1398] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:53.473001 1878 tablet_bootstrap.cc:492] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:53.473060 1879 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:53.473093 1878 ts_tablet_manager.cc:1367] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:53.473534 1878 raft_consensus.cc:348] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.473657 1878 raft_consensus.cc:374] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.473690 1878 raft_consensus.cc:729] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.473798 1878 consensus_queue.cc:260] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.473979 1879 tablet_bootstrap.cc:654] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.474086 1878 ts_tablet_manager.cc:1398] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:53.475262 1879 tablet_bootstrap.cc:492] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:53.475397 1879 ts_tablet_manager.cc:1367] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:53.475898 1879 raft_consensus.cc:348] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.476002 1879 raft_consensus.cc:374] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:53.476043 1879 raft_consensus.cc:729] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:53.476161 1879 consensus_queue.cc:260] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.476447 1879 ts_tablet_manager.cc:1398] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:53.503623 1823 raft_consensus.cc:482] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:53.503749 1823 raft_consensus.cc:504] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.504240 1823 leader_election.cc:288] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:53.504663 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071aab00f4742629e47b7aa2d6ef730" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:53.504664 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071aab00f4742629e47b7aa2d6ef730" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:53.504802 978 raft_consensus.cc:2454] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:53.504812 1242 raft_consensus.cc:2454] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:53.505136 651 leader_election.cc:302] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:53.505370 1823 raft_consensus.cc:2781] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:53.505432 1823 raft_consensus.cc:482] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:53.505466 1823 raft_consensus.cc:3037] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.506428 1823 raft_consensus.cc:504] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.506760 1823 leader_election.cc:288] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:53.507128 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071aab00f4742629e47b7aa2d6ef730" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:53.507134 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071aab00f4742629e47b7aa2d6ef730" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:53.507258 1242 raft_consensus.cc:3037] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.507272 978 raft_consensus.cc:3037] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.508107 1242 raft_consensus.cc:2454] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:53.508122 978 raft_consensus.cc:2454] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:53.508414 651 leader_election.cc:302] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:53.508587 1823 raft_consensus.cc:2781] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:53.508677 1823 raft_consensus.cc:686] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:53.508826 1823 consensus_queue.cc:237] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.510749 436 catalog_manager.cc:5484] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:53.578811 1823 raft_consensus.cc:482] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:53.578935 1823 raft_consensus.cc:504] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.579389 1823 leader_election.cc:288] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:53.579807 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00800c590d72473da73a658b54bf7d77" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:53.579854 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00800c590d72473da73a658b54bf7d77" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:53.579952 978 raft_consensus.cc:2454] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:53.580015 1242 raft_consensus.cc:2454] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:53.580272 649 leader_election.cc:302] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:53.580549 1823 raft_consensus.cc:2781] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:53.580606 1823 raft_consensus.cc:482] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:53.580658 1823 raft_consensus.cc:3037] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.581562 1823 raft_consensus.cc:504] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.581938 1823 leader_election.cc:288] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:53.582288 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00800c590d72473da73a658b54bf7d77" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:53.582334 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00800c590d72473da73a658b54bf7d77" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:53.582425 1242 raft_consensus.cc:3037] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.582456 978 raft_consensus.cc:3037] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.583238 978 raft_consensus.cc:2454] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:53.583506 1242 raft_consensus.cc:2454] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:53.583523 649 leader_election.cc:302] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:53.583828 1823 raft_consensus.cc:2781] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:53.583917 1823 raft_consensus.cc:686] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:53.584076 1823 consensus_queue.cc:237] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:53.586014 436 catalog_manager.cc:5484] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:53.637005 1632 raft_consensus.cc:482] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:53.637183 1632 raft_consensus.cc:504] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.637596 1632 leader_election.cc:288] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:53.638111 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082691bb568745b382415f1380cf1383" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:53.638099 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082691bb568745b382415f1380cf1383" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:53.638273 847 raft_consensus.cc:2454] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:53.638304 583 raft_consensus.cc:2454] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:53.638633 1047 leader_election.cc:302] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:53.638816 1632 raft_consensus.cc:2781] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:53.638886 1632 raft_consensus.cc:482] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:53.638926 1632 raft_consensus.cc:3037] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.640159 1632 raft_consensus.cc:504] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.640514 1632 leader_election.cc:288] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:53.640890 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082691bb568745b382415f1380cf1383" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:53.640918 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082691bb568745b382415f1380cf1383" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:53.641016 583 raft_consensus.cc:3037] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.641036 847 raft_consensus.cc:3037] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:53.641834 847 raft_consensus.cc:2454] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:53.641845 583 raft_consensus.cc:2454] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:53.642148 1047 leader_election.cc:302] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:53.642315 1632 raft_consensus.cc:2781] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:53.642513 1632 raft_consensus.cc:686] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:53.642669 1632 consensus_queue.cc:237] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:53.644515 436 catalog_manager.cc:5484] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:53.813280 1889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:53.813596 1889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:53.817098 1889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 01:58:53.906219 978 raft_consensus.cc:1261] T e071aab00f4742629e47b7aa2d6ef730 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.906674 1463 consensus_queue.cc:1035] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.907836 847 raft_consensus.cc:1261] T 082691bb568745b382415f1380cf1383 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.908442 583 raft_consensus.cc:1261] T 082691bb568745b382415f1380cf1383 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.908955 1887 consensus_queue.cc:1035] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.909827 1887 consensus_queue.cc:1035] T 082691bb568745b382415f1380cf1383 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.910259 1242 raft_consensus.cc:1261] T e071aab00f4742629e47b7aa2d6ef730 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:53.915911 1823 consensus_queue.cc:1035] T e071aab00f4742629e47b7aa2d6ef730 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.917155 1243 raft_consensus.cc:1261] T 00800c590d72473da73a658b54bf7d77 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:58:53.918931 1847 mvcc.cc:205] Tried to move back new op lower bound from 7023417482882187264 to 7023417481801809920. Current Snapshot: MvccSnapshot[applied={T|T < 7023417482882187264}]
I20240503 01:58:53.920194 979 raft_consensus.cc:1261] T 00800c590d72473da73a658b54bf7d77 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:58:53.921730 1463 consensus_queue.cc:1035] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:53.925439 1823 consensus_queue.cc:1035] T 00800c590d72473da73a658b54bf7d77 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.593 ms
time per row: 0.0622966 ms
I20240503 01:58:53.989204 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:53.995146 1091 tablet_service.cc:1467] Processing CreateTablet for tablet ef7bc5a7a4914f2fb62261d42ab7201c (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.995252 562 tablet_service.cc:1467] Processing CreateTablet for tablet ef7bc5a7a4914f2fb62261d42ab7201c (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.995522 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7bc5a7a4914f2fb62261d42ab7201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.995609 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7bc5a7a4914f2fb62261d42ab7201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.997355 1949 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:53.997455 1950 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:53.997581 1223 tablet_service.cc:1467] Processing CreateTablet for tablet d57c925949774d699eca47d89eea4d4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.997877 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57c925949774d699eca47d89eea4d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.998601 1950 tablet_bootstrap.cc:654] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.998651 827 tablet_service.cc:1467] Processing CreateTablet for tablet ef7bc5a7a4914f2fb62261d42ab7201c (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.998922 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7bc5a7a4914f2fb62261d42ab7201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:53.998922 1949 tablet_bootstrap.cc:654] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:53.999511 959 tablet_service.cc:1467] Processing CreateTablet for tablet d57c925949774d699eca47d89eea4d4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:53.999833 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57c925949774d699eca47d89eea4d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.000003 694 tablet_service.cc:1467] Processing CreateTablet for tablet d57c925949774d699eca47d89eea4d4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.000303 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57c925949774d699eca47d89eea4d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.000684 1953 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:54.001428 1954 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:54.001564 1953 tablet_bootstrap.cc:654] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.002252 826 tablet_service.cc:1467] Processing CreateTablet for tablet fcbf4866ccb2412a8f71f12160a16e68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.002425 1954 tablet_bootstrap.cc:654] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.002497 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbf4866ccb2412a8f71f12160a16e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.002593 1953 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:54.002681 1953 ts_tablet_manager.cc:1367] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:54.003082 562 tablet_service.cc:1467] Processing CreateTablet for tablet fcbf4866ccb2412a8f71f12160a16e68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.003096 1953 raft_consensus.cc:348] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.003206 1953 raft_consensus.cc:374] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.003238 1953 raft_consensus.cc:729] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.003355 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbf4866ccb2412a8f71f12160a16e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.003377 1953 consensus_queue.cc:260] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.003429 1949 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:54.003520 1949 ts_tablet_manager.cc:1367] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:58:54.003712 1953 ts_tablet_manager.cc:1398] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.003996 1949 raft_consensus.cc:348] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.004101 1949 raft_consensus.cc:374] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.004133 1949 raft_consensus.cc:729] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.004289 1949 consensus_queue.cc:260] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.004666 1949 ts_tablet_manager.cc:1398] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.005229 1957 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:54.005270 1954 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:54.005357 1954 ts_tablet_manager.cc:1367] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:54.005782 1954 raft_consensus.cc:348] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.005905 1954 raft_consensus.cc:374] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.005935 1954 raft_consensus.cc:729] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.006170 1957 tablet_bootstrap.cc:654] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.006211 1954 consensus_queue.cc:260] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.006444 1950 tablet_bootstrap.cc:492] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:54.006513 1954 ts_tablet_manager.cc:1398] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.006544 1950 ts_tablet_manager.cc:1367] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:58:54.006659 1954 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:54.007041 1950 raft_consensus.cc:348] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.007138 1950 raft_consensus.cc:374] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.007167 1950 raft_consensus.cc:729] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.007305 1950 consensus_queue.cc:260] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.007679 1954 tablet_bootstrap.cc:654] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.007684 1957 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:54.007762 1957 ts_tablet_manager.cc:1367] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:54.008185 1957 raft_consensus.cc:348] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.008285 1957 raft_consensus.cc:374] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.008316 1957 raft_consensus.cc:729] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.008453 1957 consensus_queue.cc:260] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.008734 1957 ts_tablet_manager.cc:1398] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:54.009810 1950 ts_tablet_manager.cc:1398] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:54.010229 1950 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:54.011276 1950 tablet_bootstrap.cc:654] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.012002 1091 tablet_service.cc:1467] Processing CreateTablet for tablet fcbf4866ccb2412a8f71f12160a16e68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fd9ed73018114a4a979b32584ef6f283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.012281 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbf4866ccb2412a8f71f12160a16e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.012399 1955 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:54.013312 1955 tablet_bootstrap.cc:654] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.013904 1950 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:54.014034 1950 ts_tablet_manager.cc:1367] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:54.014104 1954 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:54.014231 1954 ts_tablet_manager.cc:1367] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 01:58:54.014344 1955 tablet_bootstrap.cc:492] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:54.014432 1955 ts_tablet_manager.cc:1367] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:54.014619 1950 raft_consensus.cc:348] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.014732 1950 raft_consensus.cc:374] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.014689 1954 raft_consensus.cc:348] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.014762 1950 raft_consensus.cc:729] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.014765 1949 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:54.014789 1954 raft_consensus.cc:374] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.014822 1954 raft_consensus.cc:729] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.014891 1955 raft_consensus.cc:348] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.014984 1955 raft_consensus.cc:374] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.015014 1955 raft_consensus.cc:729] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.015030 1954 consensus_queue.cc:260] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.015074 1950 consensus_queue.cc:260] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.015146 1955 consensus_queue.cc:260] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.015373 1950 ts_tablet_manager.cc:1398] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.015408 1954 ts_tablet_manager.cc:1398] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:54.015416 1955 ts_tablet_manager.cc:1398] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.016333 1949 tablet_bootstrap.cc:654] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.018239 1949 tablet_bootstrap.cc:492] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:54.018329 1949 ts_tablet_manager.cc:1367] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:54.018800 1949 raft_consensus.cc:348] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.018942 1949 raft_consensus.cc:374] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.018976 1949 raft_consensus.cc:729] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.019107 1949 consensus_queue.cc:260] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.019424 1949 ts_tablet_manager.cc:1398] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.156780 1775 raft_consensus.cc:482] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.156934 1775 raft_consensus.cc:504] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.157415 1775 leader_election.cc:288] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:54.157827 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57c925949774d699eca47d89eea4d4f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:54.157814 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57c925949774d699eca47d89eea4d4f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:54.157958 1242 raft_consensus.cc:2454] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:54.157958 715 raft_consensus.cc:2454] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:54.158298 915 leader_election.cc:302] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:54.158475 1775 raft_consensus.cc:2781] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.158532 1775 raft_consensus.cc:482] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.158581 1775 raft_consensus.cc:3037] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.159407 1775 raft_consensus.cc:504] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.159780 1775 leader_election.cc:288] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:54.160161 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57c925949774d699eca47d89eea4d4f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:54.160267 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57c925949774d699eca47d89eea4d4f" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:54.160319 1242 raft_consensus.cc:3037] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.160406 715 raft_consensus.cc:3037] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.161178 715 raft_consensus.cc:2454] T d57c925949774d699eca47d89eea4d4f P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:54.161311 1242 raft_consensus.cc:2454] T d57c925949774d699eca47d89eea4d4f P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:54.161465 913 leader_election.cc:302] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:54.161669 1775 raft_consensus.cc:2781] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.161852 1775 raft_consensus.cc:686] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Running, Role: LEADER
I20240503 01:58:54.162011 1775 consensus_queue.cc:237] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:54.163905 441 catalog_manager.cc:5484] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9686ff2d42428d81510426857a12 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "4f8d9686ff2d42428d81510426857a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:54.315356 1632 raft_consensus.cc:482] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.315503 1632 raft_consensus.cc:504] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.315991 1632 leader_election.cc:288] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.316485 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbf4866ccb2412a8f71f12160a16e68" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:54.316495 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbf4866ccb2412a8f71f12160a16e68" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:54.316633 583 raft_consensus.cc:2454] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:54.316648 847 raft_consensus.cc:2454] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:54.316954 1047 leader_election.cc:302] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.317133 1632 raft_consensus.cc:2781] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.317327 1632 raft_consensus.cc:482] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.317406 1632 raft_consensus.cc:3037] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.318388 1632 raft_consensus.cc:504] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.318738 1632 leader_election.cc:288] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.319093 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbf4866ccb2412a8f71f12160a16e68" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:54.319231 583 raft_consensus.cc:3037] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.319216 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbf4866ccb2412a8f71f12160a16e68" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:54.319331 847 raft_consensus.cc:3037] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.320353 583 raft_consensus.cc:2454] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:54.320374 847 raft_consensus.cc:2454] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:54.320680 1047 leader_election.cc:302] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.320835 1632 raft_consensus.cc:2781] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.321066 1632 raft_consensus.cc:686] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:54.321305 1632 consensus_queue.cc:237] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.323418 441 catalog_manager.cc:5484] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:54.417402 1632 raft_consensus.cc:482] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.417533 1632 raft_consensus.cc:504] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.417960 1632 leader_election.cc:288] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.418385 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7bc5a7a4914f2fb62261d42ab7201c" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:54.418443 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7bc5a7a4914f2fb62261d42ab7201c" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:54.418519 583 raft_consensus.cc:2454] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:54.418614 847 raft_consensus.cc:2454] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:54.418857 1047 leader_election.cc:302] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.419068 1632 raft_consensus.cc:2781] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.419138 1632 raft_consensus.cc:482] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.419194 1632 raft_consensus.cc:3037] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.440466 1632 raft_consensus.cc:504] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.441107 1632 leader_election.cc:288] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.441386 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7bc5a7a4914f2fb62261d42ab7201c" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:54.441535 583 raft_consensus.cc:3037] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.441617 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7bc5a7a4914f2fb62261d42ab7201c" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:54.441764 847 raft_consensus.cc:3037] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.442400 583 raft_consensus.cc:2454] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:54.442692 847 raft_consensus.cc:2454] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:54.442727 1047 leader_election.cc:302] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.442899 1632 raft_consensus.cc:2781] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.442983 1632 raft_consensus.cc:686] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:54.443313 1632 consensus_queue.cc:237] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.445263 441 catalog_manager.cc:5484] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:54.586427 1969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:54.586719 1969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:54.590060 1969 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 01:58:54.621865 1775 consensus_queue.cc:1035] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.628466 1775 consensus_queue.cc:1035] T d57c925949774d699eca47d89eea4d4f P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.686991 847 raft_consensus.cc:1261] T ef7bc5a7a4914f2fb62261d42ab7201c P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:54.687604 1632 consensus_queue.cc:1035] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.688285 583 raft_consensus.cc:1261] T ef7bc5a7a4914f2fb62261d42ab7201c P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:54.688737 1632 consensus_queue.cc:1035] T ef7bc5a7a4914f2fb62261d42ab7201c P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.697440 2006 mvcc.cc:205] Tried to move back new op lower bound from 7023417486073655296 to 7023417485081534464. Current Snapshot: MvccSnapshot[applied={T|T < 7023417486073655296}]
I20240503 01:58:54.698964 581 raft_consensus.cc:1261] T fcbf4866ccb2412a8f71f12160a16e68 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:54.699682 846 raft_consensus.cc:1261] T fcbf4866ccb2412a8f71f12160a16e68 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:54.700999 1887 consensus_queue.cc:1035] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.702006 1887 consensus_queue.cc:1035] T fcbf4866ccb2412a8f71f12160a16e68 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:54.723127 2011 mvcc.cc:205] Tried to move back new op lower bound from 7023417486119460864 to 7023417484581703680. Current Snapshot: MvccSnapshot[applied={T|T < 7023417486119460864}]
INSERT report
rows total: 2000
time total: 101.074 ms
time per row: 0.0505368 ms
I20240503 01:58:54.746732 433 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:54.754910 694 tablet_service.cc:1467] Processing CreateTablet for tablet 111d4f597e7948ffb7be130fe4bd11c9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.754945 959 tablet_service.cc:1467] Processing CreateTablet for tablet 111d4f597e7948ffb7be130fe4bd11c9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.755232 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111d4f597e7948ffb7be130fe4bd11c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.755278 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111d4f597e7948ffb7be130fe4bd11c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.755831 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 111d4f597e7948ffb7be130fe4bd11c9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.756091 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111d4f597e7948ffb7be130fe4bd11c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.756076 562 tablet_service.cc:1467] Processing CreateTablet for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.756348 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.757907 2030 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:54.758574 1222 tablet_service.cc:1467] Processing CreateTablet for tablet d0e9a6698791408e974b5816489c8dd4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.758801 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e9a6698791408e974b5816489c8dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.759117 2030 tablet_bootstrap.cc:654] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.759570 694 tablet_service.cc:1467] Processing CreateTablet for tablet d0e9a6698791408e974b5816489c8dd4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.759811 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e9a6698791408e974b5816489c8dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.759950 826 tablet_service.cc:1467] Processing CreateTablet for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.760226 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.760939 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.761197 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.762190 2032 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:54.762670 959 tablet_service.cc:1467] Processing CreateTablet for tablet d0e9a6698791408e974b5816489c8dd4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9762a4c061074f5dabc908f74daa4629]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.762948 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e9a6698791408e974b5816489c8dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.764029 2032 tablet_bootstrap.cc:654] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.766151 2032 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:54.766186 2033 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:54.766331 2032 ts_tablet_manager.cc:1367] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:54.766880 2032 raft_consensus.cc:348] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.767127 2033 tablet_bootstrap.cc:654] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.767139 2032 raft_consensus.cc:374] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.767413 2032 raft_consensus.cc:729] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.768184 2032 consensus_queue.cc:260] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.768966 2030 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:54.769134 2030 ts_tablet_manager.cc:1367] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 01:58:54.769654 2030 raft_consensus.cc:348] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.769851 2030 raft_consensus.cc:374] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.769927 2030 raft_consensus.cc:729] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.770126 2030 consensus_queue.cc:260] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.770503 2030 ts_tablet_manager.cc:1398] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:54.770969 2030 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:54.771338 2032 ts_tablet_manager.cc:1398] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:58:54.772318 2030 tablet_bootstrap.cc:654] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.773018 2033 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:54.773474 2033 ts_tablet_manager.cc:1367] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:58:54.773975 2030 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:54.774068 2030 ts_tablet_manager.cc:1367] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:54.774009 2033 raft_consensus.cc:348] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.774350 2033 raft_consensus.cc:374] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.774392 2033 raft_consensus.cc:729] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.774499 2030 raft_consensus.cc:348] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.774530 2033 consensus_queue.cc:260] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.774602 2030 raft_consensus.cc:374] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.774632 2030 raft_consensus.cc:729] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.774742 2030 consensus_queue.cc:260] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.774988 2030 ts_tablet_manager.cc:1398] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:54.775346 2033 ts_tablet_manager.cc:1398] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:54.777863 2031 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:54.778970 2031 tablet_bootstrap.cc:654] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.779268 2029 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:54.780241 2029 tablet_bootstrap.cc:654] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.780843 2034 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:54.782006 2034 tablet_bootstrap.cc:654] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.782123 2029 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:54.782235 2029 ts_tablet_manager.cc:1367] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:54.782720 2029 raft_consensus.cc:348] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.782946 2029 raft_consensus.cc:374] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.782994 2029 raft_consensus.cc:729] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.783145 2029 consensus_queue.cc:260] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.783250 2034 tablet_bootstrap.cc:492] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:54.783336 2034 ts_tablet_manager.cc:1367] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:54.783516 2029 ts_tablet_manager.cc:1398] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.783774 2029 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:54.783772 2034 raft_consensus.cc:348] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.783867 2034 raft_consensus.cc:374] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.783902 2034 raft_consensus.cc:729] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.784010 2034 consensus_queue.cc:260] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.784260 2034 ts_tablet_manager.cc:1398] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.786453 2029 tablet_bootstrap.cc:654] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.788563 2029 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:54.788662 2029 ts_tablet_manager.cc:1367] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:54.789078 2029 raft_consensus.cc:348] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.789184 2029 raft_consensus.cc:374] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.789207 2029 raft_consensus.cc:729] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.789321 2029 consensus_queue.cc:260] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.789577 2029 ts_tablet_manager.cc:1398] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:54.804044 2031 tablet_bootstrap.cc:492] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:54.804167 2031 ts_tablet_manager.cc:1367] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 01:58:54.804847 2031 raft_consensus.cc:348] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.804975 2031 raft_consensus.cc:374] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.804997 2031 raft_consensus.cc:729] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.805150 2031 consensus_queue.cc:260] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.805464 2031 ts_tablet_manager.cc:1398] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:58:54.805694 2031 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:54.806713 2031 tablet_bootstrap.cc:654] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.807991 2031 tablet_bootstrap.cc:492] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:54.808082 2031 ts_tablet_manager.cc:1367] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:54.808547 2031 raft_consensus.cc:348] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.808645 2031 raft_consensus.cc:374] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.808678 2031 raft_consensus.cc:729] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.808820 2031 consensus_queue.cc:260] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.809149 2031 ts_tablet_manager.cc:1398] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:54.809595 1927 raft_consensus.cc:482] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.809717 1927 raft_consensus.cc:504] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.810189 1927 leader_election.cc:288] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:54.810480 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111d4f597e7948ffb7be130fe4bd11c9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:54.810599 1242 raft_consensus.cc:2454] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:54.810578 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111d4f597e7948ffb7be130fe4bd11c9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:54.810721 978 raft_consensus.cc:2454] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:54.810967 651 leader_election.cc:302] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:54.811198 1927 raft_consensus.cc:2781] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.811245 1927 raft_consensus.cc:482] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.811271 1927 raft_consensus.cc:3037] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.812382 1927 raft_consensus.cc:504] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.812757 1927 leader_election.cc:288] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:54.813033 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111d4f597e7948ffb7be130fe4bd11c9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:54.813156 1242 raft_consensus.cc:3037] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.813198 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111d4f597e7948ffb7be130fe4bd11c9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:54.813333 978 raft_consensus.cc:3037] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.813952 1242 raft_consensus.cc:2454] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:54.814110 978 raft_consensus.cc:2454] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:54.814258 651 leader_election.cc:302] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:54.814424 1927 raft_consensus.cc:2781] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.814646 1927 raft_consensus.cc:686] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:54.814803 1927 consensus_queue.cc:237] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:54.817734 439 catalog_manager.cc:5484] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:54.821511 1783 raft_consensus.cc:482] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:54.821645 1783 raft_consensus.cc:504] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.822021 1783 leader_election.cc:288] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.822378 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47eef4d8b93f4a43b6cffd1d4d8cc10e" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:54.822412 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47eef4d8b93f4a43b6cffd1d4d8cc10e" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:54.822510 1109 raft_consensus.cc:2454] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:54.822551 847 raft_consensus.cc:2454] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:54.822793 517 leader_election.cc:302] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.822983 1783 raft_consensus.cc:2781] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:54.823042 1783 raft_consensus.cc:482] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:54.823077 1783 raft_consensus.cc:3037] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.823952 1783 raft_consensus.cc:504] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.824265 1783 leader_election.cc:288] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:54.824581 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47eef4d8b93f4a43b6cffd1d4d8cc10e" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:54.824644 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47eef4d8b93f4a43b6cffd1d4d8cc10e" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:54.824700 1109 raft_consensus.cc:3037] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.824774 847 raft_consensus.cc:3037] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.825467 1109 raft_consensus.cc:2454] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:54.825553 847 raft_consensus.cc:2454] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:54.825780 517 leader_election.cc:302] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:54.825947 1783 raft_consensus.cc:2781] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.826084 1783 raft_consensus.cc:686] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:54.826238 1783 consensus_queue.cc:237] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:54.828763 439 catalog_manager.cc:5484] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:55.031528 2007 raft_consensus.cc:482] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:55.031704 2007 raft_consensus.cc:504] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.032243 2007 leader_election.cc:288] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:55.032681 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e9a6698791408e974b5816489c8dd4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:55.032868 715 raft_consensus.cc:2454] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:55.032871 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e9a6698791408e974b5816489c8dd4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:55.033067 1242 raft_consensus.cc:2454] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:55.033191 913 leader_election.cc:302] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, 4f8d9686ff2d42428d81510426857a12; no voters:
I20240503 01:58:55.033396 2007 raft_consensus.cc:2781] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:55.033460 2007 raft_consensus.cc:482] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:55.033488 2007 raft_consensus.cc:3037] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.034428 2007 raft_consensus.cc:504] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.034838 2007 leader_election.cc:288] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:55.035295 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e9a6698791408e974b5816489c8dd4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:55.035297 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e9a6698791408e974b5816489c8dd4" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:55.035440 1242 raft_consensus.cc:3037] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.035440 715 raft_consensus.cc:3037] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.036541 1242 raft_consensus.cc:2454] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:55.036542 715 raft_consensus.cc:2454] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:55.036895 915 leader_election.cc:302] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:55.037073 2007 raft_consensus.cc:2781] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.037165 2007 raft_consensus.cc:686] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Running, Role: LEADER
I20240503 01:58:55.037343 2007 consensus_queue.cc:237] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.039256 439 catalog_manager.cc:5484] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9686ff2d42428d81510426857a12 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "4f8d9686ff2d42428d81510426857a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:55.179401 2049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.179750 2049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.183136 2049 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 01:58:55.223145 1783 consensus_queue.cc:1035] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.227196 1783 consensus_queue.cc:1035] T 47eef4d8b93f4a43b6cffd1d4d8cc10e P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.287338 1242 raft_consensus.cc:1261] T 111d4f597e7948ffb7be130fe4bd11c9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.287798 978 raft_consensus.cc:1261] T 111d4f597e7948ffb7be130fe4bd11c9 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.288427 2044 consensus_queue.cc:1035] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.288950 2044 consensus_queue.cc:1035] T 111d4f597e7948ffb7be130fe4bd11c9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.299177 2085 mvcc.cc:205] Tried to move back new op lower bound from 7023417488530403328 to 7023417486602928128. Current Snapshot: MvccSnapshot[applied={T|T < 7023417488530403328}]
I20240503 01:58:55.300257 1242 raft_consensus.cc:1261] T d0e9a6698791408e974b5816489c8dd4 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.300926 715 raft_consensus.cc:1261] T d0e9a6698791408e974b5816489c8dd4 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.301012 2007 consensus_queue.cc:1035] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.301669 2007 consensus_queue.cc:1035] T d0e9a6698791408e974b5816489c8dd4 P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.355737 2096 mvcc.cc:205] Tried to move back new op lower bound from 7023417488530403328 to 7023417486602928128. Current Snapshot: MvccSnapshot[applied={T|T < 7023417488530403328 or (T in {7023417488530403328})}]
INSERT report
rows total: 2000
time total: 130.802 ms
time per row: 0.0654008 ms
I20240503 01:58:55.378415 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:55.385213 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 7d072eb887f84381815a96b11379e15b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.385288 562 tablet_service.cc:1467] Processing CreateTablet for tablet 7d072eb887f84381815a96b11379e15b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.385589 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d072eb887f84381815a96b11379e15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.385591 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d072eb887f84381815a96b11379e15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.385944 826 tablet_service.cc:1467] Processing CreateTablet for tablet 7d072eb887f84381815a96b11379e15b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.386073 694 tablet_service.cc:1467] Processing CreateTablet for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.386199 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d072eb887f84381815a96b11379e15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.386337 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.388361 563 tablet_service.cc:1467] Processing CreateTablet for tablet 19d2814cdb7e4f7592bb1b610de5565b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.388370 1222 tablet_service.cc:1467] Processing CreateTablet for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.388589 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d2814cdb7e4f7592bb1b610de5565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.388589 959 tablet_service.cc:1467] Processing CreateTablet for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.388653 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.388872 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a4df3fe9d24a76a7121f7a5cfdf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.388826 827 tablet_service.cc:1467] Processing CreateTablet for tablet 19d2814cdb7e4f7592bb1b610de5565b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.389070 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d2814cdb7e4f7592bb1b610de5565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.391767 2115 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:55.392037 2116 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:55.391757 2114 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:55.392361 2111 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:55.393025 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 19d2814cdb7e4f7592bb1b610de5565b (DEFAULT_TABLE table=rebalance_test_table_10 [id=66e52d4c1ffd437c8e611663384b8c24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.393168 2114 tablet_bootstrap.cc:654] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.393168 2115 tablet_bootstrap.cc:654] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.393407 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d2814cdb7e4f7592bb1b610de5565b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.393579 2112 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:55.393412 2111 tablet_bootstrap.cc:654] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.394618 2113 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:55.394841 2112 tablet_bootstrap.cc:654] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.394887 2116 tablet_bootstrap.cc:654] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.396107 2112 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:55.396162 2116 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:55.396200 2112 ts_tablet_manager.cc:1367] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:55.396265 2116 ts_tablet_manager.cc:1367] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:55.396263 2113 tablet_bootstrap.cc:654] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.396669 2112 raft_consensus.cc:348] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.396669 2116 raft_consensus.cc:348] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.396780 2116 raft_consensus.cc:374] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.396780 2112 raft_consensus.cc:374] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.396813 2112 raft_consensus.cc:729] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.396813 2116 raft_consensus.cc:729] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.396946 2116 consensus_queue.cc:260] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.396946 2112 consensus_queue.cc:260] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.397244 2116 ts_tablet_manager.cc:1398] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.397240 2112 ts_tablet_manager.cc:1398] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.397405 2112 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:55.398133 2114 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:55.398249 2114 ts_tablet_manager.cc:1367] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:55.398283 2112 tablet_bootstrap.cc:654] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.398485 2115 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:55.398577 2115 ts_tablet_manager.cc:1367] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:55.398766 2114 raft_consensus.cc:348] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.398887 2114 raft_consensus.cc:374] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.398923 2114 raft_consensus.cc:729] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.399078 2114 consensus_queue.cc:260] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.399067 2115 raft_consensus.cc:348] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.399174 2115 raft_consensus.cc:374] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.399207 2115 raft_consensus.cc:729] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.399331 2115 consensus_queue.cc:260] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.399401 2114 ts_tablet_manager.cc:1398] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.399612 2114 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:55.399632 2111 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:55.399711 2111 ts_tablet_manager.cc:1367] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:55.400391 2111 raft_consensus.cc:348] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.400521 2111 raft_consensus.cc:374] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.400559 2111 raft_consensus.cc:729] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.400692 2111 consensus_queue.cc:260] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.400935 2112 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:55.401021 2112 ts_tablet_manager.cc:1367] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:55.401026 2111 ts_tablet_manager.cc:1398] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.401113 2114 tablet_bootstrap.cc:654] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.401202 2111 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:55.401458 2112 raft_consensus.cc:348] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.401561 2112 raft_consensus.cc:374] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.401590 2112 raft_consensus.cc:729] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.401695 2112 consensus_queue.cc:260] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.401757 2115 ts_tablet_manager.cc:1398] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:55.401955 2112 ts_tablet_manager.cc:1398] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.402148 2111 tablet_bootstrap.cc:654] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.404096 2113 tablet_bootstrap.cc:492] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:55.404270 2113 ts_tablet_manager.cc:1367] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:58:55.404873 2113 raft_consensus.cc:348] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.405088 2113 raft_consensus.cc:374] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.405179 2114 tablet_bootstrap.cc:492] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:55.405193 2113 raft_consensus.cc:729] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.405283 2114 ts_tablet_manager.cc:1367] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:55.405489 2113 consensus_queue.cc:260] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.405681 2111 tablet_bootstrap.cc:492] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:55.405836 2111 ts_tablet_manager.cc:1367] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:55.405786 2114 raft_consensus.cc:348] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.405889 2114 raft_consensus.cc:374] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.405925 2114 raft_consensus.cc:729] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.406067 2114 consensus_queue.cc:260] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.406420 2114 ts_tablet_manager.cc:1398] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.406618 2111 raft_consensus.cc:348] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.406776 2111 raft_consensus.cc:374] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.406857 2111 raft_consensus.cc:729] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.407069 2111 consensus_queue.cc:260] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.407496 2111 ts_tablet_manager.cc:1398] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:55.408381 2113 ts_tablet_manager.cc:1398] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:55.445600 2090 raft_consensus.cc:482] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:55.445745 2090 raft_consensus.cc:504] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.446205 2090 leader_election.cc:288] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:55.446838 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d2814cdb7e4f7592bb1b610de5565b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:55.446835 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d2814cdb7e4f7592bb1b610de5565b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:55.447006 1110 raft_consensus.cc:2454] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:55.447022 847 raft_consensus.cc:2454] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:55.447374 517 leader_election.cc:302] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:55.447576 2090 raft_consensus.cc:2781] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:55.447639 2090 raft_consensus.cc:482] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:55.447695 2090 raft_consensus.cc:3037] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.449132 2090 raft_consensus.cc:504] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.449524 2090 leader_election.cc:288] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:55.449967 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d2814cdb7e4f7592bb1b610de5565b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:55.450009 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d2814cdb7e4f7592bb1b610de5565b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:55.450140 847 raft_consensus.cc:3037] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.450142 1110 raft_consensus.cc:3037] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.451125 847 raft_consensus.cc:2454] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:55.451494 1110 raft_consensus.cc:2454] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:55.451534 517 leader_election.cc:302] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:55.451889 2090 raft_consensus.cc:2781] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.452387 2090 raft_consensus.cc:686] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:55.452570 2090 consensus_queue.cc:237] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } }
I20240503 01:58:55.455044 437 catalog_manager.cc:5484] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:55.461987 2007 raft_consensus.cc:482] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:55.462229 2007 raft_consensus.cc:504] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.462637 2007 leader_election.cc:288] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:55.463054 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a4df3fe9d24a76a7121f7a5cfdf58e" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c" is_pre_election: true
I20240503 01:58:55.463063 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a4df3fe9d24a76a7121f7a5cfdf58e" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:55.463191 715 raft_consensus.cc:2454] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:55.463205 1242 raft_consensus.cc:2454] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 0.
I20240503 01:58:55.463483 915 leader_election.cc:302] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:55.463668 2007 raft_consensus.cc:2781] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:55.463732 2007 raft_consensus.cc:482] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:55.463774 2007 raft_consensus.cc:3037] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.533566 2007 raft_consensus.cc:504] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.534098 2007 leader_election.cc:288] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Requested vote from peers 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130:43979), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:55.534694 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a4df3fe9d24a76a7121f7a5cfdf58e" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:55.534799 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a4df3fe9d24a76a7121f7a5cfdf58e" candidate_uuid: "4f8d9686ff2d42428d81510426857a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123bf566427645e9b94b1c01ebf9cd4c"
I20240503 01:58:55.534853 1242 raft_consensus.cc:3037] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.534925 715 raft_consensus.cc:3037] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.535996 715 raft_consensus.cc:2454] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:55.536396 1242 raft_consensus.cc:2454] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8d9686ff2d42428d81510426857a12 in term 1.
I20240503 01:58:55.536825 915 leader_election.cc:302] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8d9686ff2d42428d81510426857a12, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:55.537117 2007 raft_consensus.cc:2781] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.537217 2007 raft_consensus.cc:686] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Running, Role: LEADER
I20240503 01:58:55.537380 2007 consensus_queue.cc:237] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } }
I20240503 01:58:55.539888 437 catalog_manager.cc:5484] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8d9686ff2d42428d81510426857a12 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "4f8d9686ff2d42428d81510426857a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:55.636029 2090 raft_consensus.cc:482] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:55.636189 2090 raft_consensus.cc:504] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.636704 2090 leader_election.cc:288] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:55.637250 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d072eb887f84381815a96b11379e15b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57" is_pre_election: true
I20240503 01:58:55.637265 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d072eb887f84381815a96b11379e15b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:55.637424 847 raft_consensus.cc:2454] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:55.637424 1110 raft_consensus.cc:2454] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 0.
I20240503 01:58:55.637734 517 leader_election.cc:302] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:55.637943 2090 raft_consensus.cc:2781] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:55.638001 2090 raft_consensus.cc:482] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:55.638060 2090 raft_consensus.cc:3037] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.638878 2090 raft_consensus.cc:504] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.639240 2090 leader_election.cc:288] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Requested vote from peers 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133:37453), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:55.639719 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d072eb887f84381815a96b11379e15b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a60c6be59b44b88af9c886a1b1ef57"
I20240503 01:58:55.639731 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d072eb887f84381815a96b11379e15b" candidate_uuid: "158b7d66c1e8460aae25035f17afdb14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:55.639858 847 raft_consensus.cc:3037] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.639884 1110 raft_consensus.cc:3037] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.640678 847 raft_consensus.cc:2454] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:55.640708 1110 raft_consensus.cc:2454] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158b7d66c1e8460aae25035f17afdb14 in term 1.
I20240503 01:58:55.641021 517 leader_election.cc:302] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:55.641186 2090 raft_consensus.cc:2781] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.641278 2090 raft_consensus.cc:686] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [term 1 LEADER]: Becoming Leader. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Running, Role: LEADER
I20240503 01:58:55.641434 2090 consensus_queue.cc:237] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:55.643347 437 catalog_manager.cc:5484] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "158b7d66c1e8460aae25035f17afdb14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
W20240503 01:58:55.814204 2129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.814504 2129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.818095 2129 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 01:58:55.916271 1110 raft_consensus.cc:1261] T 7d072eb887f84381815a96b11379e15b P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.916689 847 raft_consensus.cc:1261] T 7d072eb887f84381815a96b11379e15b P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.916746 1783 consensus_queue.cc:1035] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.917098 2090 consensus_queue.cc:1035] T 7d072eb887f84381815a96b11379e15b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.920500 1111 raft_consensus.cc:1261] T 19d2814cdb7e4f7592bb1b610de5565b P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.920809 846 raft_consensus.cc:1261] T 19d2814cdb7e4f7592bb1b610de5565b P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 158b7d66c1e8460aae25035f17afdb14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.922184 2090 consensus_queue.cc:1035] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.922582 2090 consensus_queue.cc:1035] T 19d2814cdb7e4f7592bb1b610de5565b P 158b7d66c1e8460aae25035f17afdb14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.926110 1242 raft_consensus.cc:1261] T a1a4df3fe9d24a76a7121f7a5cfdf58e P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.926584 715 raft_consensus.cc:1261] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Refusing update from remote peer 4f8d9686ff2d42428d81510426857a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:55.927047 1964 consensus_queue.cc:1035] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:55.927291 2007 consensus_queue.cc:1035] T a1a4df3fe9d24a76a7121f7a5cfdf58e P 4f8d9686ff2d42428d81510426857a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.781 ms
time per row: 0.0648905 ms
I20240503 01:58:55.999133 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58988:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:56.005067 959 tablet_service.cc:1467] Processing CreateTablet for tablet 29f23c90267f4f4c9acb4297df67f084 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.005156 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 29f23c90267f4f4c9acb4297df67f084 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.005396 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f23c90267f4f4c9acb4297df67f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.005491 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f23c90267f4f4c9acb4297df67f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.005610 694 tablet_service.cc:1467] Processing CreateTablet for tablet 29f23c90267f4f4c9acb4297df67f084 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.005872 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f23c90267f4f4c9acb4297df67f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.007234 2191 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:56.007848 562 tablet_service.cc:1467] Processing CreateTablet for tablet 6264ca58d48b4a73ab6e14b8f23a46a7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.008034 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 86b770cc82fe4c53ba222535029a24b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.008087 2192 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:56.008159 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6264ca58d48b4a73ab6e14b8f23a46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.008280 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b770cc82fe4c53ba222535029a24b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.009052 2192 tablet_bootstrap.cc:654] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.010295 2192 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:56.010464 2192 ts_tablet_manager.cc:1367] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:56.010751 694 tablet_service.cc:1467] Processing CreateTablet for tablet 86b770cc82fe4c53ba222535029a24b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.011035 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b770cc82fe4c53ba222535029a24b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.011004 959 tablet_service.cc:1467] Processing CreateTablet for tablet 86b770cc82fe4c53ba222535029a24b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.011144 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 6264ca58d48b4a73ab6e14b8f23a46a7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.011250 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b770cc82fe4c53ba222535029a24b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.011413 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6264ca58d48b4a73ab6e14b8f23a46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.011605 2191 tablet_bootstrap.cc:654] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.011879 827 tablet_service.cc:1467] Processing CreateTablet for tablet 6264ca58d48b4a73ab6e14b8f23a46a7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=189649d27b404c289af70eae40525bd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.012147 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6264ca58d48b4a73ab6e14b8f23a46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.012732 2191 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:56.012811 2191 ts_tablet_manager.cc:1367] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 01:58:56.013305 2191 raft_consensus.cc:348] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.013418 2191 raft_consensus.cc:374] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.013458 2191 raft_consensus.cc:729] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.013568 2191 consensus_queue.cc:260] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.013847 2191 ts_tablet_manager.cc:1398] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:56.014006 2191 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12: Bootstrap starting.
I20240503 01:58:56.013916 2192 raft_consensus.cc:348] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.014041 2192 raft_consensus.cc:374] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.014073 2192 raft_consensus.cc:729] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.014199 2192 consensus_queue.cc:260] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.014297 2197 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57: Bootstrap starting.
I20240503 01:58:56.014937 2191 tablet_bootstrap.cc:654] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.015570 2192 ts_tablet_manager.cc:1398] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:58:56.015744 2192 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d: Bootstrap starting.
I20240503 01:58:56.015897 2197 tablet_bootstrap.cc:654] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.016749 2192 tablet_bootstrap.cc:654] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.017025 2193 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:56.017246 2191 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12: No bootstrap required, opened a new log
I20240503 01:58:56.017341 2191 ts_tablet_manager.cc:1367] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:56.017781 2191 raft_consensus.cc:348] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.017925 2191 raft_consensus.cc:374] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.017961 2191 raft_consensus.cc:729] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8d9686ff2d42428d81510426857a12, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.018083 2191 consensus_queue.cc:260] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.018322 2193 tablet_bootstrap.cc:654] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.018369 2191 ts_tablet_manager.cc:1398] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:56.019520 2198 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c: Bootstrap starting.
I20240503 01:58:56.020238 2197 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57: No bootstrap required, opened a new log
I20240503 01:58:56.020560 2197 ts_tablet_manager.cc:1367] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:56.020676 2192 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d: No bootstrap required, opened a new log
I20240503 01:58:56.020778 2192 ts_tablet_manager.cc:1367] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:56.020810 2198 tablet_bootstrap.cc:654] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.021116 2197 raft_consensus.cc:348] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.021258 2197 raft_consensus.cc:374] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.021199 2192 raft_consensus.cc:348] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.021306 2192 raft_consensus.cc:374] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.021380 2192 raft_consensus.cc:729] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22eb38cc62f46d58c4bcdfb38ae776d, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.021644 2192 consensus_queue.cc:260] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.021767 2193 tablet_bootstrap.cc:492] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:56.021870 2193 ts_tablet_manager.cc:1367] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:58:56.021962 2197 raft_consensus.cc:729] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.022075 2192 ts_tablet_manager.cc:1398] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:56.022130 2198 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c: No bootstrap required, opened a new log
I20240503 01:58:56.022187 2197 consensus_queue.cc:260] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.022262 2198 ts_tablet_manager.cc:1367] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:56.022281 2195 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14: Bootstrap starting.
I20240503 01:58:56.022560 2197 ts_tablet_manager.cc:1398] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:56.023113 2193 raft_consensus.cc:348] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.023387 2198 raft_consensus.cc:348] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.023716 2198 raft_consensus.cc:374] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.023802 2198 raft_consensus.cc:729] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77ccf223a649f590c7c2a7b632768c, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.024025 2198 consensus_queue.cc:260] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.025265 2195 tablet_bootstrap.cc:654] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.025698 2193 raft_consensus.cc:374] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.025815 2193 raft_consensus.cc:729] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.025861 2198 ts_tablet_manager.cc:1398] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:58:56.025998 2193 consensus_queue.cc:260] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.026302 2193 ts_tablet_manager.cc:1398] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:58:56.026505 2193 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c: Bootstrap starting.
I20240503 01:58:56.027578 2193 tablet_bootstrap.cc:654] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.028350 2195 tablet_bootstrap.cc:492] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14: No bootstrap required, opened a new log
I20240503 01:58:56.028519 2195 ts_tablet_manager.cc:1367] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:58:56.028826 2193 tablet_bootstrap.cc:492] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c: No bootstrap required, opened a new log
I20240503 01:58:56.028913 2193 ts_tablet_manager.cc:1367] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:58:56.029158 2195 raft_consensus.cc:348] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.029268 2195 raft_consensus.cc:374] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.029291 2195 raft_consensus.cc:729] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158b7d66c1e8460aae25035f17afdb14, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.029332 2193 raft_consensus.cc:348] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.029407 2195 consensus_queue.cc:260] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.029443 2193 raft_consensus.cc:374] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.029479 2193 raft_consensus.cc:729] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.029608 2193 consensus_queue.cc:260] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.029713 2195 ts_tablet_manager.cc:1398] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:58:56.029908 2193 ts_tablet_manager.cc:1398] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:58:56.068147 2140 raft_consensus.cc:482] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:56.068316 2140 raft_consensus.cc:504] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.069072 2140 leader_election.cc:288] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:56.069496 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6264ca58d48b4a73ab6e14b8f23a46a7" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14" is_pre_election: true
I20240503 01:58:56.069514 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6264ca58d48b4a73ab6e14b8f23a46a7" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c" is_pre_election: true
I20240503 01:58:56.069666 583 raft_consensus.cc:2454] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:56.069667 846 raft_consensus.cc:2454] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 0.
I20240503 01:58:56.070003 1045 leader_election.cc:302] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a60c6be59b44b88af9c886a1b1ef57, cf77ccf223a649f590c7c2a7b632768c; no voters:
I20240503 01:58:56.070189 2140 raft_consensus.cc:2781] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:56.070250 2140 raft_consensus.cc:482] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:56.070286 2140 raft_consensus.cc:3037] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.071223 2140 raft_consensus.cc:504] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.071609 2140 leader_election.cc:288] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Requested vote from peers 158b7d66c1e8460aae25035f17afdb14 (127.0.99.129:43283), cf77ccf223a649f590c7c2a7b632768c (127.0.99.131:41289)
I20240503 01:58:56.072046 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6264ca58d48b4a73ab6e14b8f23a46a7" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158b7d66c1e8460aae25035f17afdb14"
I20240503 01:58:56.072044 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6264ca58d48b4a73ab6e14b8f23a46a7" candidate_uuid: "24a60c6be59b44b88af9c886a1b1ef57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf77ccf223a649f590c7c2a7b632768c"
I20240503 01:58:56.072194 846 raft_consensus.cc:3037] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.072194 583 raft_consensus.cc:3037] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.073231 846 raft_consensus.cc:2454] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:56.073231 583 raft_consensus.cc:2454] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a60c6be59b44b88af9c886a1b1ef57 in term 1.
I20240503 01:58:56.073618 1047 leader_election.cc:302] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158b7d66c1e8460aae25035f17afdb14, 24a60c6be59b44b88af9c886a1b1ef57; no voters:
I20240503 01:58:56.074018 2140 raft_consensus.cc:2781] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.074139 2140 raft_consensus.cc:686] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [term 1 LEADER]: Becoming Leader. State: Replica: 24a60c6be59b44b88af9c886a1b1ef57, State: Running, Role: LEADER
I20240503 01:58:56.074298 2140 consensus_queue.cc:237] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } }
I20240503 01:58:56.076584 432 catalog_manager.cc:5484] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a60c6be59b44b88af9c886a1b1ef57 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "24a60c6be59b44b88af9c886a1b1ef57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a60c6be59b44b88af9c886a1b1ef57" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:56.094028 2044 raft_consensus.cc:482] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:56.094156 2044 raft_consensus.cc:504] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.094552 2044 leader_election.cc:288] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:56.094913 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b770cc82fe4c53ba222535029a24b9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:56.094918 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b770cc82fe4c53ba222535029a24b9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:56.095050 1242 raft_consensus.cc:2454] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:56.095058 978 raft_consensus.cc:2454] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:56.095353 651 leader_election.cc:302] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:56.095511 2044 raft_consensus.cc:2781] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:56.095588 2044 raft_consensus.cc:482] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:56.095626 2044 raft_consensus.cc:3037] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.096457 2044 raft_consensus.cc:504] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.096752 2044 leader_election.cc:288] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495), b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471)
I20240503 01:58:56.097064 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b770cc82fe4c53ba222535029a24b9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:56.097110 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b770cc82fe4c53ba222535029a24b9" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:56.097218 978 raft_consensus.cc:3037] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.097229 1242 raft_consensus.cc:3037] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.097983 1242 raft_consensus.cc:2454] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:56.098037 978 raft_consensus.cc:2454] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:56.098302 651 leader_election.cc:302] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:56.098475 2044 raft_consensus.cc:2781] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.098686 2044 raft_consensus.cc:686] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:56.098841 2044 consensus_queue.cc:237] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } }
I20240503 01:58:56.100634 432 catalog_manager.cc:5484] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20240503 01:58:56.198222 2044 raft_consensus.cc:482] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:56.198386 2044 raft_consensus.cc:504] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.198834 2044 leader_election.cc:288] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:56.199301 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f23c90267f4f4c9acb4297df67f084" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12" is_pre_election: true
I20240503 01:58:56.199308 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f23c90267f4f4c9acb4297df67f084" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" is_pre_election: true
I20240503 01:58:56.199441 1242 raft_consensus.cc:2454] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:56.199462 978 raft_consensus.cc:2454] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 0.
I20240503 01:58:56.199770 651 leader_election.cc:302] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:56.199932 2044 raft_consensus.cc:2781] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:56.199986 2044 raft_consensus.cc:482] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:56.200018 2044 raft_consensus.cc:3037] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.200805 2044 raft_consensus.cc:504] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.201126 2044 leader_election.cc:288] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Requested vote from peers b22eb38cc62f46d58c4bcdfb38ae776d (127.0.99.134:37471), 4f8d9686ff2d42428d81510426857a12 (127.0.99.132:40495)
I20240503 01:58:56.201474 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f23c90267f4f4c9acb4297df67f084" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d"
I20240503 01:58:56.201536 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f23c90267f4f4c9acb4297df67f084" candidate_uuid: "123bf566427645e9b94b1c01ebf9cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8d9686ff2d42428d81510426857a12"
I20240503 01:58:56.201581 1242 raft_consensus.cc:3037] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.201669 978 raft_consensus.cc:3037] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.202383 1242 raft_consensus.cc:2454] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:56.202445 978 raft_consensus.cc:2454] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123bf566427645e9b94b1c01ebf9cd4c in term 1.
I20240503 01:58:56.202651 651 leader_election.cc:302] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123bf566427645e9b94b1c01ebf9cd4c, b22eb38cc62f46d58c4bcdfb38ae776d; no voters:
I20240503 01:58:56.202818 2044 raft_consensus.cc:2781] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.202898 2044 raft_consensus.cc:686] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 123bf566427645e9b94b1c01ebf9cd4c, State: Running, Role: LEADER
I20240503 01:58:56.203025 2044 consensus_queue.cc:237] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } }
I20240503 01:58:56.204857 432 catalog_manager.cc:5484] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 123bf566427645e9b94b1c01ebf9cd4c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "123bf566427645e9b94b1c01ebf9cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123bf566427645e9b94b1c01ebf9cd4c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43979 } health_report { overall_health: HEALTHY } } }
W20240503 01:58:56.326053 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:56.326321 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:56.329603 2210 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 01:58:56.426086 978 raft_consensus.cc:1261] T 29f23c90267f4f4c9acb4297df67f084 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.426611 2044 consensus_queue.cc:1035] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.428326 1242 raft_consensus.cc:1261] T 29f23c90267f4f4c9acb4297df67f084 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.429247 2044 consensus_queue.cc:1035] T 29f23c90267f4f4c9acb4297df67f084 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.430384 582 raft_consensus.cc:1261] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 158b7d66c1e8460aae25035f17afdb14 [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.431043 846 raft_consensus.cc:1261] T 6264ca58d48b4a73ab6e14b8f23a46a7 P cf77ccf223a649f590c7c2a7b632768c [term 1 FOLLOWER]: Refusing update from remote peer 24a60c6be59b44b88af9c886a1b1ef57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.431530 2140 consensus_queue.cc:1035] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158b7d66c1e8460aae25035f17afdb14" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.431946 2140 consensus_queue.cc:1035] T 6264ca58d48b4a73ab6e14b8f23a46a7 P 24a60c6be59b44b88af9c886a1b1ef57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf77ccf223a649f590c7c2a7b632768c" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.447491 1243 raft_consensus.cc:1261] T 86b770cc82fe4c53ba222535029a24b9 P b22eb38cc62f46d58c4bcdfb38ae776d [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.447896 979 raft_consensus.cc:1261] T 86b770cc82fe4c53ba222535029a24b9 P 4f8d9686ff2d42428d81510426857a12 [term 1 FOLLOWER]: Refusing update from remote peer 123bf566427645e9b94b1c01ebf9cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:58:56.447999 2044 consensus_queue.cc:1035] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22eb38cc62f46d58c4bcdfb38ae776d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.448732 2044 consensus_queue.cc:1035] T 86b770cc82fe4c53ba222535029a24b9 P 123bf566427645e9b94b1c01ebf9cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8d9686ff2d42428d81510426857a12" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:56.481868 2246 mvcc.cc:205] Tried to move back new op lower bound from 7023417493283184640 to 7023417491862441984. Current Snapshot: MvccSnapshot[applied={T|T < 7023417493283184640 or (T in {7023417493283184640})}]
INSERT report
rows total: 2000
time total: 124.885 ms
time per row: 0.0624425 ms
I20240503 01:58:56.728699 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:56.731948 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:56.732702 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:56.734539 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:56.735427 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:56.740213 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:56.879194 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:56.879186 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:56.882337 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:56.897599 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:56.898149 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:56.898274 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.046083 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.048902 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:57.049046 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.052165 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.057683 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:57.075394 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:57.184939 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:57.187342 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:57.191795 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.196668 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.206027 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:57.206805 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.321388 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:57.323485 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.325420 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.325439 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:57.333951 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:57.334741 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.478583 694 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:58:57.479243 562 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:58:57.488987 1223 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:58:57.490226 1091 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.491161 827 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:58:57.497963 959 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48d6008349a34c919262a7c21e448fff | 127.0.99.190:34331 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
123bf566427645e9b94b1c01ebf9cd4c | 127.0.99.130:43979 | HEALTHY | /A | 9 | 0
158b7d66c1e8460aae25035f17afdb14 | 127.0.99.129:43283 | HEALTHY | /A | 6 | 0
24a60c6be59b44b88af9c886a1b1ef57 | 127.0.99.133:37453 | HEALTHY | /C | 7 | 0
4f8d9686ff2d42428d81510426857a12 | 127.0.99.132:40495 | HEALTHY | /B | 5 | 0
b22eb38cc62f46d58c4bcdfb38ae776d | 127.0.99.134:37471 | HEALTHY | /C | 5 | 0
cf77ccf223a649f590c7c2a7b632768c | 127.0.99.131:41289 | HEALTHY | /B | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/A | 2
/C | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43283
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43979
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41289
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:40495
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:37453
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37471
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43283
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43979
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41289
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:40495
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:37453
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37471
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35145 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:58:57.535998 398 log_verifier.cc:126] Checking tablet 00800c590d72473da73a658b54bf7d77
I20240503 01:58:57.564931 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 00800c590d72473da73a658b54bf7d77
I20240503 01:58:57.565060 398 log_verifier.cc:126] Checking tablet 04a6b21333ec49eca8e66bfadec8f57a
I20240503 01:58:57.596598 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04a6b21333ec49eca8e66bfadec8f57a
I20240503 01:58:57.596666 398 log_verifier.cc:126] Checking tablet 082691bb568745b382415f1380cf1383
I20240503 01:58:57.626643 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 082691bb568745b382415f1380cf1383
I20240503 01:58:57.626739 398 log_verifier.cc:126] Checking tablet 0f0a5bb25be642dda808fde15d59dec9
I20240503 01:58:57.657238 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0f0a5bb25be642dda808fde15d59dec9
I20240503 01:58:57.657311 398 log_verifier.cc:126] Checking tablet 111d4f597e7948ffb7be130fe4bd11c9
I20240503 01:58:57.685032 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 111d4f597e7948ffb7be130fe4bd11c9
I20240503 01:58:57.685119 398 log_verifier.cc:126] Checking tablet 19d2814cdb7e4f7592bb1b610de5565b
I20240503 01:58:57.713824 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 19d2814cdb7e4f7592bb1b610de5565b
I20240503 01:58:57.713894 398 log_verifier.cc:126] Checking tablet 29f23c90267f4f4c9acb4297df67f084
I20240503 01:58:57.743036 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 29f23c90267f4f4c9acb4297df67f084
I20240503 01:58:57.743130 398 log_verifier.cc:126] Checking tablet 2b2aaee1132c40638fae59ad5f985056
I20240503 01:58:57.773317 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2b2aaee1132c40638fae59ad5f985056
I20240503 01:58:57.773406 398 log_verifier.cc:126] Checking tablet 40de9f5424f04345a31ddbcff2f5f45c
I20240503 01:58:57.803658 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 40de9f5424f04345a31ddbcff2f5f45c
I20240503 01:58:57.803738 398 log_verifier.cc:126] Checking tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e
I20240503 01:58:57.833155 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 47eef4d8b93f4a43b6cffd1d4d8cc10e
I20240503 01:58:57.833233 398 log_verifier.cc:126] Checking tablet 50b248b22783495a856627db92d92550
I20240503 01:58:57.863713 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 50b248b22783495a856627db92d92550
I20240503 01:58:57.863790 398 log_verifier.cc:126] Checking tablet 559fbdb613924b4ab7740c0560d9fe8f
I20240503 01:58:57.892947 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 559fbdb613924b4ab7740c0560d9fe8f
I20240503 01:58:57.893035 398 log_verifier.cc:126] Checking tablet 57d0b13b23c7421ea802353cbaad45a7
I20240503 01:58:57.922144 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 57d0b13b23c7421ea802353cbaad45a7
I20240503 01:58:57.922220 398 log_verifier.cc:126] Checking tablet 6264ca58d48b4a73ab6e14b8f23a46a7
I20240503 01:58:57.951736 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6264ca58d48b4a73ab6e14b8f23a46a7
I20240503 01:58:57.951834 398 log_verifier.cc:126] Checking tablet 7905b8572e194a87bb263f629b311f93
I20240503 01:58:57.980700 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7905b8572e194a87bb263f629b311f93
I20240503 01:58:57.980778 398 log_verifier.cc:126] Checking tablet 7d072eb887f84381815a96b11379e15b
I20240503 01:58:58.011265 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d072eb887f84381815a96b11379e15b
I20240503 01:58:58.011346 398 log_verifier.cc:126] Checking tablet 84225e096ac143ebb59cffd163f2d151
I20240503 01:58:58.041042 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 84225e096ac143ebb59cffd163f2d151
I20240503 01:58:58.041119 398 log_verifier.cc:126] Checking tablet 86b770cc82fe4c53ba222535029a24b9
I20240503 01:58:58.070323 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 86b770cc82fe4c53ba222535029a24b9
I20240503 01:58:58.070415 398 log_verifier.cc:126] Checking tablet 9fcf27fda05442ed9fcb87769517b9fe
I20240503 01:58:58.100934 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9fcf27fda05442ed9fcb87769517b9fe
I20240503 01:58:58.101014 398 log_verifier.cc:126] Checking tablet a191cae8ecdd46ce8f49eb11fc967449
I20240503 01:58:58.130447 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a191cae8ecdd46ce8f49eb11fc967449
I20240503 01:58:58.130530 398 log_verifier.cc:126] Checking tablet a1a4df3fe9d24a76a7121f7a5cfdf58e
I20240503 01:58:58.160931 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a1a4df3fe9d24a76a7121f7a5cfdf58e
I20240503 01:58:58.161003 398 log_verifier.cc:126] Checking tablet ba7691f939444a5ea03af2c66d4dc6ef
I20240503 01:58:58.189877 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ba7691f939444a5ea03af2c66d4dc6ef
I20240503 01:58:58.189953 398 log_verifier.cc:126] Checking tablet bb2933a9f17641a7a835a348e153da13
I20240503 01:58:58.219417 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bb2933a9f17641a7a835a348e153da13
I20240503 01:58:58.219503 398 log_verifier.cc:126] Checking tablet bd3010b22936401d8d368e758e93154e
I20240503 01:58:58.249871 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bd3010b22936401d8d368e758e93154e
I20240503 01:58:58.249936 398 log_verifier.cc:126] Checking tablet c22514cbdd5b48b890e702b7b32e8910
I20240503 01:58:58.280330 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c22514cbdd5b48b890e702b7b32e8910
I20240503 01:58:58.280400 398 log_verifier.cc:126] Checking tablet cccfe866337246b7882b876903570d0b
I20240503 01:58:58.309579 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cccfe866337246b7882b876903570d0b
I20240503 01:58:58.309674 398 log_verifier.cc:126] Checking tablet d0e9a6698791408e974b5816489c8dd4
I20240503 01:58:58.338016 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d0e9a6698791408e974b5816489c8dd4
I20240503 01:58:58.338091 398 log_verifier.cc:126] Checking tablet d2a5bae6b9644d68bec77be04d9d3c45
I20240503 01:58:58.370010 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d2a5bae6b9644d68bec77be04d9d3c45
I20240503 01:58:58.370077 398 log_verifier.cc:126] Checking tablet d57c925949774d699eca47d89eea4d4f
I20240503 01:58:58.399093 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d57c925949774d699eca47d89eea4d4f
I20240503 01:58:58.399206 398 log_verifier.cc:126] Checking tablet e071aab00f4742629e47b7aa2d6ef730
I20240503 01:58:58.427110 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e071aab00f4742629e47b7aa2d6ef730
I20240503 01:58:58.427193 398 log_verifier.cc:126] Checking tablet ecc2fc58a14d4379831e4a94189c2693
I20240503 01:58:58.456662 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ecc2fc58a14d4379831e4a94189c2693
I20240503 01:58:58.456748 398 log_verifier.cc:126] Checking tablet edace0c39397435c929e99df6f2a5f0b
I20240503 01:58:58.486630 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet edace0c39397435c929e99df6f2a5f0b
I20240503 01:58:58.486721 398 log_verifier.cc:126] Checking tablet ef7bc5a7a4914f2fb62261d42ab7201c
I20240503 01:58:58.516374 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ef7bc5a7a4914f2fb62261d42ab7201c
I20240503 01:58:58.516453 398 log_verifier.cc:126] Checking tablet f504aaf5826842988eacf0e4d13de3f6
I20240503 01:58:58.546913 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f504aaf5826842988eacf0e4d13de3f6
I20240503 01:58:58.546998 398 log_verifier.cc:126] Checking tablet fcbf4866ccb2412a8f71f12160a16e68
I20240503 01:58:58.576743 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fcbf4866ccb2412a8f71f12160a16e68
I20240503 01:58:58.576822 398 log_verifier.cc:126] Checking tablet fcd37670627f480a8883d3c448304ff4
I20240503 01:58:58.606396 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fcd37670627f480a8883d3c448304ff4
I20240503 01:58:58.619525 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 501
I20240503 01:58:58.626981 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 632
I20240503 01:58:58.634593 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 765
I20240503 01:58:58.641891 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 896
I20240503 01:58:58.648726 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 1028
I20240503 01:58:58.655141 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 1161
I20240503 01:58:58.662087 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 410
2024-05-03T01:58:58Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (14659 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (14659 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 01:58:58.783455 398 test_util.cc:268] Using random seed: 270364890
I20240503 01:58:58.784217 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:58.784268 398 ts_itest-base.cc:113] --------------
I20240503 01:58:58.784281 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:58:58.784291 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:58.784300 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:58Z Disabled control of system clock
I20240503 01:58:58.792508 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42231
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42231
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:58:58.905141 2544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:58.905452 2544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:58.905534 2544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:58.908890 2544 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:58.909020 2544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:58.909051 2544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:58.909085 2544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:58.909125 2544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:58.913221 2544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:58.914278 2544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:58.915531 2544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:58.925081 2549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:58.925899 2544 server_base.cc:1034] running on GCE node
W20240503 01:58:58.926949 2550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:58.927985 2552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:58.928642 2544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:58.929702 2544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:58.930867 2544 hybrid_clock.cc:648] HybridClock initialized: now 1714701538930854 us; error 41 us; skew 500 ppm
I20240503 01:58:58.933292 2544 webserver.cc:460] Webserver started at http://127.0.99.190:43367/ using document root <none> and password file <none>
I20240503 01:58:58.933943 2544 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:58.934011 2544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:58.934226 2544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:58.936255 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6b5760b7e4274eafbaece331292495d6"
format_stamp: "Formatted at 2024-05-03 01:58:58 on dist-test-slave-pvfz"
I20240503 01:58:58.936795 2544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6b5760b7e4274eafbaece331292495d6"
format_stamp: "Formatted at 2024-05-03 01:58:58 on dist-test-slave-pvfz"
I20240503 01:58:58.940564 2544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:58.942996 2558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:58.944202 2544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:58.944337 2544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6b5760b7e4274eafbaece331292495d6"
format_stamp: "Formatted at 2024-05-03 01:58:58 on dist-test-slave-pvfz"
I20240503 01:58:58.944439 2544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:58.972046 2544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:58.972736 2544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:58.972905 2544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:58.979373 2544 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42231
I20240503 01:58:58.979384 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42231 every 8 connection(s)
I20240503 01:58:58.980461 2544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:58.983197 2611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.988411 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 2544
I20240503 01:58:58.988516 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:58.988632 2611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6: Bootstrap starting.
I20240503 01:58:58.991237 2611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:58.992205 2611 log.cc:824] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:58.994324 2611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6: No bootstrap required, opened a new log
I20240503 01:58:58.997670 2611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } }
I20240503 01:58:58.997882 2611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:58.997942 2611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5760b7e4274eafbaece331292495d6, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.998411 2611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } }
I20240503 01:58:58.998559 2611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:58.998618 2611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:58.998737 2611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:58.999656 2611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } }
I20240503 01:58:58.999975 2611 leader_election.cc:302] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b5760b7e4274eafbaece331292495d6; no voters:
I20240503 01:58:59.000216 2611 leader_election.cc:288] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:59.000347 2616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:59.000950 2616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6b5760b7e4274eafbaece331292495d6, State: Running, Role: LEADER
I20240503 01:58:59.001286 2616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } }
I20240503 01:58:59.001384 2611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:59.002825 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b5760b7e4274eafbaece331292495d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } } }
I20240503 01:58:59.002849 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b5760b7e4274eafbaece331292495d6. Latest consensus state: current_term: 1 leader_uuid: "6b5760b7e4274eafbaece331292495d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5760b7e4274eafbaece331292495d6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42231 } } }
I20240503 01:58:59.002941 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:59.002965 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:59.003373 2628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:59.006592 2628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:59.012408 2628 catalog_manager.cc:1347] Generated new cluster ID: ac4390a0fb2e442b8cd0a062d8f32d84
I20240503 01:58:59.012501 2628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:59.036301 2628 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:59.037693 2628 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:59.048907 2628 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6b5760b7e4274eafbaece331292495d6: Generated new TSK 0
I20240503 01:58:59.049844 2628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:59.061239 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:58:59.175086 2635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.175356 2635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:59.175433 2635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.178730 2635 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:59.178828 2635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:59.178936 2635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:59.182760 2635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.2635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:59.183831 2635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:59.185026 2635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:59.194288 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.194949 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.196060 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:59.197075 2635 server_base.cc:1034] running on GCE node
I20240503 01:58:59.197569 2635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:59.198240 2635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:59.199425 2635 hybrid_clock.cc:648] HybridClock initialized: now 1714701539199397 us; error 46 us; skew 500 ppm
I20240503 01:58:59.201606 2635 webserver.cc:460] Webserver started at http://127.0.99.129:35851/ using document root <none> and password file <none>
I20240503 01:58:59.202239 2635 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:59.202306 2635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:59.202525 2635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:59.204391 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d5e6747cb87b411ab04c02cc0235cd5b"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.204880 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d5e6747cb87b411ab04c02cc0235cd5b"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.208308 2635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:59.210597 2649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.211724 2635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:59.211835 2635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d5e6747cb87b411ab04c02cc0235cd5b"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.211934 2635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:59.241981 2635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:59.242683 2635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:59.242861 2635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:59.243427 2635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:59.244393 2635 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:59.244444 2635 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.244508 2635 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:59.244539 2635 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.253492 2635 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:42769
I20240503 01:58:59.253515 2762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:42769 every 8 connection(s)
I20240503 01:58:59.254525 2635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:59.257855 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 2635
I20240503 01:58:59.257983 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:59.260133 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:59.264496 2763 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42231
I20240503 01:58:59.264741 2763 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:59.265318 2763 heartbeater.cc:511] Master 127.0.99.190:42231 requested a full tablet report, sending...
I20240503 01:58:59.266779 2575 ts_manager.cc:194] Registered new tserver with Master: d5e6747cb87b411ab04c02cc0235cd5b (127.0.99.129:42769)
I20240503 01:58:59.267992 2575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:44403
W20240503 01:58:59.371491 2766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.371779 2766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:59.371840 2766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.375015 2766 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:59.375087 2766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:59.375167 2766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:59.379016 2766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.2766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:59.380071 2766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:59.381206 2766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:59.390842 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.391770 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.392562 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:59.393440 2766 server_base.cc:1034] running on GCE node
I20240503 01:58:59.393903 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:59.394563 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:59.395733 2766 hybrid_clock.cc:648] HybridClock initialized: now 1714701539395704 us; error 48 us; skew 500 ppm
I20240503 01:58:59.397891 2766 webserver.cc:460] Webserver started at http://127.0.99.130:41957/ using document root <none> and password file <none>
I20240503 01:58:59.398495 2766 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:59.398562 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:59.398766 2766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:59.400722 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "afb33e4271514aefb9fb15a363c54fa7"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.401338 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "afb33e4271514aefb9fb15a363c54fa7"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.405130 2766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 01:58:59.407743 2780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.408880 2766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:58:59.409025 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "afb33e4271514aefb9fb15a363c54fa7"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.409132 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:59.436795 2766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:59.437502 2766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:59.437695 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:59.438266 2766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:59.439257 2766 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:59.439307 2766 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.439368 2766 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:59.439393 2766 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.447856 2766 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:42957
I20240503 01:58:59.447880 2893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:42957 every 8 connection(s)
I20240503 01:58:59.448951 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:59.456367 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 2766
I20240503 01:58:59.456506 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:59.458144 2894 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42231
I20240503 01:58:59.458400 2894 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:59.458606 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:59.459002 2894 heartbeater.cc:511] Master 127.0.99.190:42231 requested a full tablet report, sending...
I20240503 01:58:59.460096 2575 ts_manager.cc:194] Registered new tserver with Master: afb33e4271514aefb9fb15a363c54fa7 (127.0.99.130:42957)
I20240503 01:58:59.460656 2575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43567
W20240503 01:58:59.568821 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.569085 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:59.569132 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.572384 2897 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:59.572471 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:59.572562 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:59.576457 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.2897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:59.577477 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:59.578701 2897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:59.588008 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.588666 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.589476 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:59.590553 2897 server_base.cc:1034] running on GCE node
I20240503 01:58:59.590978 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:59.591609 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:59.592763 2897 hybrid_clock.cc:648] HybridClock initialized: now 1714701539592748 us; error 33 us; skew 500 ppm
I20240503 01:58:59.594976 2897 webserver.cc:460] Webserver started at http://127.0.99.131:36479/ using document root <none> and password file <none>
I20240503 01:58:59.595609 2897 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:59.595676 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:59.595883 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:59.597635 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9e092edb07734fc2923917143a072387"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.598152 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9e092edb07734fc2923917143a072387"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.601750 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 01:58:59.604379 2911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.605474 2897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:59.605602 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e092edb07734fc2923917143a072387"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.605713 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:59.628857 2897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:59.629540 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:59.629706 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:59.630259 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:59.631157 2897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:59.631207 2897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.631268 2897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:59.631294 2897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.639602 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:33843
I20240503 01:58:59.639626 3024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:33843 every 8 connection(s)
I20240503 01:58:59.640549 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:59.644572 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 2897
I20240503 01:58:59.644656 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:59.646744 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:59.649307 3025 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42231
I20240503 01:58:59.649557 3025 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:59.650260 3025 heartbeater.cc:511] Master 127.0.99.190:42231 requested a full tablet report, sending...
I20240503 01:58:59.651329 2575 ts_manager.cc:194] Registered new tserver with Master: 9e092edb07734fc2923917143a072387 (127.0.99.131:33843)
I20240503 01:58:59.651904 2575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:42311
W20240503 01:58:59.758496 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.758767 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:59.758813 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.762141 3028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:59.762234 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:59.762341 3028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:59.766105 3028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.3028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:59.767135 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:59.768352 3028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:59.777844 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.778487 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.779245 3033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:59.780388 3028 server_base.cc:1034] running on GCE node
I20240503 01:58:59.780862 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:59.781491 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:59.782655 3028 hybrid_clock.cc:648] HybridClock initialized: now 1714701539782632 us; error 46 us; skew 500 ppm
I20240503 01:58:59.784790 3028 webserver.cc:460] Webserver started at http://127.0.99.132:43869/ using document root <none> and password file <none>
I20240503 01:58:59.785359 3028 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:59.785420 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:59.785602 3028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:59.787359 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6f7da2112db448b9a58c8df6ce0ec276"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.787905 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6f7da2112db448b9a58c8df6ce0ec276"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.791311 3028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:59.793676 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.794708 3028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:59.794819 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6f7da2112db448b9a58c8df6ce0ec276"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.794912 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:59.823189 3028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:59.823892 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:59.824090 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:59.824656 3028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:59.825603 3028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:59.825654 3028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.825726 3028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:59.825753 3028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.834604 3028 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36527
I20240503 01:58:59.834663 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36527 every 8 connection(s)
I20240503 01:58:59.835639 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:59.843477 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3028
I20240503 01:58:59.843626 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:59.844826 3156 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42231
I20240503 01:58:59.845109 3156 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:59.845678 3156 heartbeater.cc:511] Master 127.0.99.190:42231 requested a full tablet report, sending...
I20240503 01:58:59.845726 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--builtin_ntp_servers=127.0.99.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:58:59.846685 2575 ts_manager.cc:194] Registered new tserver with Master: 6f7da2112db448b9a58c8df6ce0ec276 (127.0.99.132:36527)
I20240503 01:58:59.847252 2575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:36587
W20240503 01:58:59.960402 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:59.960657 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:59.960718 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:59.964100 3159 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:58:59.964186 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:59.964280 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:59.968143 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:59.969146 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:59.970332 3159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:59.979708 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.980325 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:59.981330 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:59.982306 3159 server_base.cc:1034] running on GCE node
I20240503 01:58:59.982748 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:59.983397 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:59.984596 3159 hybrid_clock.cc:648] HybridClock initialized: now 1714701539984572 us; error 33 us; skew 500 ppm
I20240503 01:58:59.986673 3159 webserver.cc:460] Webserver started at http://127.0.99.133:42807/ using document root <none> and password file <none>
I20240503 01:58:59.987237 3159 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:59.987299 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:59.987471 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:59.989159 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d19c4d0f5a9244568055bc2ce8ed88e8"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.989616 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d19c4d0f5a9244568055bc2ce8ed88e8"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.992913 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:59.994997 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:59.996052 3159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:59.996168 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d19c4d0f5a9244568055bc2ce8ed88e8"
format_stamp: "Formatted at 2024-05-03 01:58:59 on dist-test-slave-pvfz"
I20240503 01:58:59.996256 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:00.011989 3159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:00.012655 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:00.012822 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:00.013401 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:00.014305 3159 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:00.014367 3159 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:00.014425 3159 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:00.014451 3159 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:00.022522 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35179
I20240503 01:59:00.022559 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35179 every 8 connection(s)
I20240503 01:59:00.023468 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:00.032056 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3159
I20240503 01:59:00.032150 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:00.037375 3287 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42231
I20240503 01:59:00.037633 3287 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:00.038195 3287 heartbeater.cc:511] Master 127.0.99.190:42231 requested a full tablet report, sending...
I20240503 01:59:00.039149 2575 ts_manager.cc:194] Registered new tserver with Master: d19c4d0f5a9244568055bc2ce8ed88e8 (127.0.99.133:35179)
I20240503 01:59:00.039736 2575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:38479
I20240503 01:59:00.046422 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:59:00.056005 2575 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36552:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:59:00.074144 2959 tablet_service.cc:1467] Processing CreateTablet for tablet ad4e1d86d1e34bc680b7af92a40e0ecc (DEFAULT_TABLE table=TestTable [id=84f661ae11644287aea352522634d616]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.074144 2697 tablet_service.cc:1467] Processing CreateTablet for tablet ad4e1d86d1e34bc680b7af92a40e0ecc (DEFAULT_TABLE table=TestTable [id=84f661ae11644287aea352522634d616]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.074144 3089 tablet_service.cc:1467] Processing CreateTablet for tablet ad4e1d86d1e34bc680b7af92a40e0ecc (DEFAULT_TABLE table=TestTable [id=84f661ae11644287aea352522634d616]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:00.075080 2959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4e1d86d1e34bc680b7af92a40e0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.075084 2697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4e1d86d1e34bc680b7af92a40e0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.075083 3089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4e1d86d1e34bc680b7af92a40e0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.080510 3305 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Bootstrap starting.
I20240503 01:59:00.081116 3306 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Bootstrap starting.
I20240503 01:59:00.083092 3306 tablet_bootstrap.cc:654] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.083092 3305 tablet_bootstrap.cc:654] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.083904 3307 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Bootstrap starting.
I20240503 01:59:00.084015 3305 log.cc:824] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:00.084015 3306 log.cc:824] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:00.085850 3307 tablet_bootstrap.cc:654] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:00.085908 3305 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: No bootstrap required, opened a new log
I20240503 01:59:00.085908 3306 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: No bootstrap required, opened a new log
I20240503 01:59:00.086084 3306 ts_tablet_manager.cc:1367] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:00.086084 3305 ts_tablet_manager.cc:1367] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:00.086601 3307 log.cc:824] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:00.088682 3307 tablet_bootstrap.cc:492] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: No bootstrap required, opened a new log
I20240503 01:59:00.088878 3307 ts_tablet_manager.cc:1367] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:00.090087 3305 raft_consensus.cc:348] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.090087 3306 raft_consensus.cc:348] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.090330 3305 raft_consensus.cc:374] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.090330 3306 raft_consensus.cc:374] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.090390 3305 raft_consensus.cc:729] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e092edb07734fc2923917143a072387, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.090391 3306 raft_consensus.cc:729] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e6747cb87b411ab04c02cc0235cd5b, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.090883 3305 consensus_queue.cc:260] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.090883 3306 consensus_queue.cc:260] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.091852 3306 ts_tablet_manager.cc:1398] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 01:59:00.091852 3305 ts_tablet_manager.cc:1398] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 01:59:00.092280 2763 heartbeater.cc:503] Master 127.0.99.190:42231 was elected leader, sending a full tablet report...
I20240503 01:59:00.092309 3025 heartbeater.cc:503] Master 127.0.99.190:42231 was elected leader, sending a full tablet report...
I20240503 01:59:00.093214 3307 raft_consensus.cc:348] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.093447 3307 raft_consensus.cc:374] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:00.093509 3307 raft_consensus.cc:729] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7da2112db448b9a58c8df6ce0ec276, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.093953 3307 consensus_queue.cc:260] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.094575 3156 heartbeater.cc:503] Master 127.0.99.190:42231 was elected leader, sending a full tablet report...
I20240503 01:59:00.095508 3307 ts_tablet_manager.cc:1398] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20240503 01:59:00.141681 3026 tablet.cc:2369] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:00.256166 2764 tablet.cc:2369] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:00.281930 3311 raft_consensus.cc:482] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:00.282171 3311 raft_consensus.cc:504] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.283336 3311 leader_election.cc:288] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f7da2112db448b9a58c8df6ce0ec276 (127.0.99.132:36527), d5e6747cb87b411ab04c02cc0235cd5b (127.0.99.129:42769)
I20240503 01:59:00.287986 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4e1d86d1e34bc680b7af92a40e0ecc" candidate_uuid: "9e092edb07734fc2923917143a072387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7da2112db448b9a58c8df6ce0ec276" is_pre_election: true
I20240503 01:59:00.287981 2717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4e1d86d1e34bc680b7af92a40e0ecc" candidate_uuid: "9e092edb07734fc2923917143a072387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" is_pre_election: true
I20240503 01:59:00.288250 2717 raft_consensus.cc:2454] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e092edb07734fc2923917143a072387 in term 0.
I20240503 01:59:00.288250 3108 raft_consensus.cc:2454] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e092edb07734fc2923917143a072387 in term 0.
I20240503 01:59:00.288676 2913 leader_election.cc:302] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7da2112db448b9a58c8df6ce0ec276, 9e092edb07734fc2923917143a072387; no voters:
I20240503 01:59:00.288923 3311 raft_consensus.cc:2781] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:00.288990 3311 raft_consensus.cc:482] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:00.289031 3311 raft_consensus.cc:3037] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.289875 3311 raft_consensus.cc:504] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.290207 3311 leader_election.cc:288] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [CANDIDATE]: Term 1 election: Requested vote from peers 6f7da2112db448b9a58c8df6ce0ec276 (127.0.99.132:36527), d5e6747cb87b411ab04c02cc0235cd5b (127.0.99.129:42769)
I20240503 01:59:00.290553 2717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4e1d86d1e34bc680b7af92a40e0ecc" candidate_uuid: "9e092edb07734fc2923917143a072387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e6747cb87b411ab04c02cc0235cd5b"
I20240503 01:59:00.290553 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4e1d86d1e34bc680b7af92a40e0ecc" candidate_uuid: "9e092edb07734fc2923917143a072387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7da2112db448b9a58c8df6ce0ec276"
I20240503 01:59:00.290678 3108 raft_consensus.cc:3037] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.290678 2717 raft_consensus.cc:3037] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:00.291760 3108 raft_consensus.cc:2454] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e092edb07734fc2923917143a072387 in term 1.
I20240503 01:59:00.291760 2717 raft_consensus.cc:2454] T ad4e1d86d1e34bc680b7af92a40e0ecc P d5e6747cb87b411ab04c02cc0235cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e092edb07734fc2923917143a072387 in term 1.
I20240503 01:59:00.292099 2914 leader_election.cc:302] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e092edb07734fc2923917143a072387, d5e6747cb87b411ab04c02cc0235cd5b; no voters:
I20240503 01:59:00.292261 3311 raft_consensus.cc:2781] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:00.292451 3311 raft_consensus.cc:686] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [term 1 LEADER]: Becoming Leader. State: Replica: 9e092edb07734fc2923917143a072387, State: Running, Role: LEADER
I20240503 01:59:00.292801 3311 consensus_queue.cc:237] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } }
I20240503 01:59:00.295867 2575 catalog_manager.cc:5484] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e092edb07734fc2923917143a072387 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9e092edb07734fc2923917143a072387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e092edb07734fc2923917143a072387" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33843 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:00.301771 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:59:00.303236 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d5e6747cb87b411ab04c02cc0235cd5b to finish bootstrapping
I20240503 01:59:00.314651 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver afb33e4271514aefb9fb15a363c54fa7 to finish bootstrapping
I20240503 01:59:00.320606 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9e092edb07734fc2923917143a072387 to finish bootstrapping
I20240503 01:59:00.325265 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6f7da2112db448b9a58c8df6ce0ec276 to finish bootstrapping
I20240503 01:59:00.330390 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d19c4d0f5a9244568055bc2ce8ed88e8 to finish bootstrapping
W20240503 01:59:00.336660 3157 tablet.cc:2369] T ad4e1d86d1e34bc680b7af92a40e0ecc P 6f7da2112db448b9a58c8df6ce0ec276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:00.462651 2894 heartbeater.cc:503] Master 127.0.99.190:42231 was elected leader, sending a full tablet report...
I20240503 01:59:00.513942 3089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.515655 2828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.515774 2697 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.516407 3221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.517432 2959 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:00.605976 3221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.606014 3089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.609411 2959 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:00.611775 2828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.627086 2697 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.840858 3089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.841118 2697 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.842622 2959 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:00.843199 2828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.849886 3419 consensus_queue.cc:1035] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e6747cb87b411ab04c02cc0235cd5b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.857679 3419 consensus_queue.cc:1035] T ad4e1d86d1e34bc680b7af92a40e0ecc P 9e092edb07734fc2923917143a072387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7da2112db448b9a58c8df6ce0ec276" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:00.860433 3221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.949707 2697 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.949927 2828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.950939 3089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.955432 3221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:00.964258 2959 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:01.041770 3287 heartbeater.cc:503] Master 127.0.99.190:42231 was elected leader, sending a full tablet report...
I20240503 01:59:01.069839 2697 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:01.070470 2959 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:01.070473 3089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:01.071945 2828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:01.075781 3221 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:01.114332 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 2635
I20240503 01:59:01.120568 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 2766
I20240503 01:59:01.126778 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 2897
I20240503 01:59:01.133159 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3028
I20240503 01:59:01.138985 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3159
I20240503 01:59:01.144608 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 2544
2024-05-03T01:59:01Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2384 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2385 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240503 01:59:01.168582 398 test_util.cc:268] Using random seed: 272750017
I20240503 01:59:01.169342 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:01.169384 398 ts_itest-base.cc:113] --------------
I20240503 01:59:01.169395 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 01:59:01.169405 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 01:59:01.169416 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:01Z Disabled control of system clock
I20240503 01:59:01.177870 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33221
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:33221
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 01:59:01.291040 3505 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 01:59:01.291332 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.291383 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:01.291453 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.294628 3505 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:01.294730 3505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:01.294766 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:01.294798 3505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:01.294983 3505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:01.299050 3505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:33221
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:01.300130 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:01.301319 3505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:01.309476 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.310197 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.311336 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:01.312309 3505 server_base.cc:1034] running on GCE node
I20240503 01:59:01.312904 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:01.313927 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:01.315085 3505 hybrid_clock.cc:648] HybridClock initialized: now 1714701541315074 us; error 31 us; skew 500 ppm
I20240503 01:59:01.317135 3505 webserver.cc:460] Webserver started at http://127.0.99.190:34217/ using document root <none> and password file <none>
I20240503 01:59:01.317720 3505 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:01.317777 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:01.317981 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:01.319763 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d88debf910664f928bbd978e084af0e0"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.320233 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d88debf910664f928bbd978e084af0e0"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.323760 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:01.326117 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.327193 3505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:01.327332 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d88debf910664f928bbd978e084af0e0"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.327440 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:01.345074 3505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:01.345767 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:01.345928 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:01.352937 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:33221
I20240503 01:59:01.352952 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:33221 every 8 connection(s)
I20240503 01:59:01.353996 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:01.357008 3572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:01.362673 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0: Bootstrap starting.
I20240503 01:59:01.363960 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3505
I20240503 01:59:01.364101 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:01.365653 3572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:01.366384 3572 log.cc:824] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:01.368270 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0: No bootstrap required, opened a new log
I20240503 01:59:01.371161 3572 raft_consensus.cc:348] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } }
I20240503 01:59:01.371393 3572 raft_consensus.cc:374] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:01.371471 3572 raft_consensus.cc:729] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88debf910664f928bbd978e084af0e0, State: Initialized, Role: FOLLOWER
I20240503 01:59:01.371986 3572 consensus_queue.cc:260] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } }
I20240503 01:59:01.372136 3572 raft_consensus.cc:388] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:01.372195 3572 raft_consensus.cc:482] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:01.372304 3572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:01.373064 3572 raft_consensus.cc:504] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } }
I20240503 01:59:01.373394 3572 leader_election.cc:302] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d88debf910664f928bbd978e084af0e0; no voters:
I20240503 01:59:01.373682 3572 leader_election.cc:288] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:01.373817 3577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:01.374027 3577 raft_consensus.cc:686] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [term 1 LEADER]: Becoming Leader. State: Replica: d88debf910664f928bbd978e084af0e0, State: Running, Role: LEADER
I20240503 01:59:01.374344 3577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } }
I20240503 01:59:01.374840 3572 sys_catalog.cc:564] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:01.375938 3578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d88debf910664f928bbd978e084af0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } } }
I20240503 01:59:01.376060 3578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:01.376359 3586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:01.376907 3579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d88debf910664f928bbd978e084af0e0. Latest consensus state: current_term: 1 leader_uuid: "d88debf910664f928bbd978e084af0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88debf910664f928bbd978e084af0e0" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33221 } } }
I20240503 01:59:01.377002 3579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:01.379453 3586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:01.384738 3586 catalog_manager.cc:1347] Generated new cluster ID: d0a115321e304d2287ab009572760f43
I20240503 01:59:01.384838 3586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:01.410610 3586 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:01.411705 3586 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:01.417557 3586 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d88debf910664f928bbd978e084af0e0: Generated new TSK 0
I20240503 01:59:01.418287 3586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:01.425396 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:01.543891 3596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.544147 3596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:01.544224 3596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.547392 3596 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:01.547477 3596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:01.547617 3596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:01.551378 3596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.3596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:01.552402 3596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:01.553586 3596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:01.563024 3602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.563997 3604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.564831 3601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:01.565657 3596 server_base.cc:1034] running on GCE node
I20240503 01:59:01.566088 3596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:01.566659 3596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:01.567803 3596 hybrid_clock.cc:648] HybridClock initialized: now 1714701541567790 us; error 31 us; skew 500 ppm
I20240503 01:59:01.569844 3596 webserver.cc:460] Webserver started at http://127.0.99.129:40033/ using document root <none> and password file <none>
I20240503 01:59:01.570392 3596 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:01.570447 3596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:01.570616 3596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:01.572325 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2820cbd55ee1470880e35a8ab671563b"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.572757 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2820cbd55ee1470880e35a8ab671563b"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.576015 3596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:01.578277 3610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.579278 3596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:01.579386 3596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2820cbd55ee1470880e35a8ab671563b"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.579473 3596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:01.590852 3596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:01.591528 3596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:01.591747 3596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:01.592299 3596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:01.593206 3596 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:01.593254 3596 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.593314 3596 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:01.593340 3596 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.601879 3596 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43461
I20240503 01:59:01.601914 3723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43461 every 8 connection(s)
I20240503 01:59:01.602872 3596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:01.612105 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3596
I20240503 01:59:01.612229 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:01.614593 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:01.616070 3724 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:01.616307 3724 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:01.616969 3724 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:01.618477 3535 ts_manager.cc:194] Registered new tserver with Master: 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:01.619691 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:36669
W20240503 01:59:01.726902 3727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.727169 3727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:01.727216 3727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.730571 3727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:01.730655 3727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:01.730756 3727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:01.734709 3727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.3727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:01.735741 3727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:01.736922 3727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:01.746628 3733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.747364 3732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.748430 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:01.749329 3727 server_base.cc:1034] running on GCE node
I20240503 01:59:01.749816 3727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:01.750449 3727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:01.751621 3727 hybrid_clock.cc:648] HybridClock initialized: now 1714701541751608 us; error 33 us; skew 500 ppm
I20240503 01:59:01.753628 3727 webserver.cc:460] Webserver started at http://127.0.99.130:40883/ using document root <none> and password file <none>
I20240503 01:59:01.754220 3727 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:01.754292 3727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:01.754482 3727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:01.756366 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f929df19fdfc4df1986e990c60056733"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.756901 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f929df19fdfc4df1986e990c60056733"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.760985 3727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 01:59:01.763365 3741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.764503 3727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:01.764642 3727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f929df19fdfc4df1986e990c60056733"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.764750 3727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:01.781787 3727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:01.782462 3727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:01.782615 3727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:01.783210 3727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:01.784204 3727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:01.784252 3727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.784298 3727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:01.784332 3727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.792941 3727 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:35911
I20240503 01:59:01.792989 3854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:35911 every 8 connection(s)
I20240503 01:59:01.793891 3727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:01.801230 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3727
I20240503 01:59:01.801394 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:01.803535 3855 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:01.803826 3855 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:01.803826 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:01.804445 3855 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:01.805632 3535 ts_manager.cc:194] Registered new tserver with Master: f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:01.806466 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:42089
W20240503 01:59:01.917070 3858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:01.917430 3858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:01.917519 3858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:01.921002 3858 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:01.921089 3858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:01.921164 3858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:01.925163 3858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.3858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:01.926156 3858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:01.927357 3858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:01.937004 3863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.937722 3864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:01.938813 3866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:01.939723 3858 server_base.cc:1034] running on GCE node
I20240503 01:59:01.940196 3858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:01.940868 3858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:01.942050 3858 hybrid_clock.cc:648] HybridClock initialized: now 1714701541942033 us; error 35 us; skew 500 ppm
I20240503 01:59:01.944381 3858 webserver.cc:460] Webserver started at http://127.0.99.131:44489/ using document root <none> and password file <none>
I20240503 01:59:01.945011 3858 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:01.945072 3858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:01.945281 3858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:01.947078 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c8595f9ae8f644d6901791810d7b970e"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.947623 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c8595f9ae8f644d6901791810d7b970e"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.951195 3858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:01.953703 3872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.954892 3858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:01.955022 3858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c8595f9ae8f644d6901791810d7b970e"
format_stamp: "Formatted at 2024-05-03 01:59:01 on dist-test-slave-pvfz"
I20240503 01:59:01.955121 3858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:01.985993 3858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:01.986718 3858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:01.986893 3858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:01.987457 3858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:01.988399 3858 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:01.988449 3858 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.988514 3858 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:01.988541 3858 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:01.997277 3858 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:36705
I20240503 01:59:01.997308 3985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:36705 every 8 connection(s)
I20240503 01:59:01.998271 3858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:02.000576 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3858
I20240503 01:59:02.000700 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:02.003039 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:02.008347 3986 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:02.008653 3986 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:02.009490 3986 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:02.010780 3535 ts_manager.cc:194] Registered new tserver with Master: c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:02.011377 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:40975
W20240503 01:59:02.117199 3989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:02.117460 3989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:02.117511 3989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:02.120798 3989 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:02.120882 3989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:02.120954 3989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:02.124722 3989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.3989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:02.125720 3989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:02.126883 3989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:02.136540 3997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.137264 3995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.138442 3994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:02.139324 3989 server_base.cc:1034] running on GCE node
I20240503 01:59:02.139796 3989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:02.140409 3989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:02.141561 3989 hybrid_clock.cc:648] HybridClock initialized: now 1714701542141542 us; error 37 us; skew 500 ppm
I20240503 01:59:02.143915 3989 webserver.cc:460] Webserver started at http://127.0.99.132:35363/ using document root <none> and password file <none>
I20240503 01:59:02.144614 3989 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:02.144688 3989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:02.144888 3989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:02.146721 3989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7b5c836ac65b442e808f46fb2d267d2c"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.147234 3989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7b5c836ac65b442e808f46fb2d267d2c"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.150962 3989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 01:59:02.153436 4003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.154662 3989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:02.154810 3989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7b5c836ac65b442e808f46fb2d267d2c"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.154919 3989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:02.181847 3989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:02.182546 3989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:02.182723 3989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:02.183297 3989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:02.184335 3989 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:02.184386 3989 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.184443 3989 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:02.184475 3989 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.193125 3989 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35901
I20240503 01:59:02.193158 4116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35901 every 8 connection(s)
I20240503 01:59:02.194137 3989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:02.199515 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 3989
I20240503 01:59:02.199688 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:02.202028 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:02.203971 4117 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:02.204232 4117 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:02.204846 4117 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:02.205991 3535 ts_manager.cc:194] Registered new tserver with Master: 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901)
I20240503 01:59:02.206588 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:53981
W20240503 01:59:02.313894 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:02.314163 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:02.314241 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:02.317543 4120 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:02.317637 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:02.317745 4120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:02.321648 4120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.4120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:02.322695 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:02.323953 4120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:02.333308 4126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.334309 4128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.335095 4125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:02.336447 4120 server_base.cc:1034] running on GCE node
I20240503 01:59:02.336918 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:02.337558 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:02.338722 4120 hybrid_clock.cc:648] HybridClock initialized: now 1714701542338709 us; error 32 us; skew 500 ppm
I20240503 01:59:02.340857 4120 webserver.cc:460] Webserver started at http://127.0.99.133:36207/ using document root <none> and password file <none>
I20240503 01:59:02.341442 4120 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:02.341502 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:02.341707 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:02.343472 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c6f646ec3edd4e298f16122028bb0ef7"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.344026 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c6f646ec3edd4e298f16122028bb0ef7"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.347718 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:02.350188 4134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.351310 4120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:02.351439 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c6f646ec3edd4e298f16122028bb0ef7"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.351586 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:02.381126 4120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:02.381843 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:02.381992 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:02.382565 4120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:02.383515 4120 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:02.383594 4120 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.383644 4120 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:02.383666 4120 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.391916 4120 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41793
I20240503 01:59:02.391963 4247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41793 every 8 connection(s)
I20240503 01:59:02.392905 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:02.398901 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 4120
I20240503 01:59:02.399026 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:02.401377 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:02.402151 4248 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:02.402420 4248 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:02.403087 4248 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:02.404220 3535 ts_manager.cc:194] Registered new tserver with Master: c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:02.404831 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35275
W20240503 01:59:02.515708 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:02.515964 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:02.516009 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:02.519366 4251 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:02.519451 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:02.519579 4251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:02.523393 4251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.4251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:02.524466 4251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:02.525643 4251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:02.534907 4256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.535895 4259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.536748 4257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:02.537823 4251 server_base.cc:1034] running on GCE node
I20240503 01:59:02.538260 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:02.538885 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:02.540056 4251 hybrid_clock.cc:648] HybridClock initialized: now 1714701542540036 us; error 33 us; skew 500 ppm
I20240503 01:59:02.542152 4251 webserver.cc:460] Webserver started at http://127.0.99.134:46193/ using document root <none> and password file <none>
I20240503 01:59:02.542753 4251 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:02.542820 4251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:02.543005 4251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:02.544814 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7a367744629a43edb0df92463a498258"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.545336 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7a367744629a43edb0df92463a498258"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.549075 4251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 01:59:02.551426 4265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.552484 4251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:02.552608 4251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7a367744629a43edb0df92463a498258"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.552711 4251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:02.571966 4251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:02.572664 4251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:02.572829 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:02.573381 4251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:02.574309 4251 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:02.574362 4251 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.574419 4251 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:02.574450 4251 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.583475 4251 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42343
I20240503 01:59:02.583519 4378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42343 every 8 connection(s)
I20240503 01:59:02.584570 4251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:02.587991 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 4251
I20240503 01:59:02.588124 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:02.590389 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:02.594348 4379 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:02.594590 4379 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:02.595178 4379 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:02.596400 3535 ts_manager.cc:194] Registered new tserver with Master: 7a367744629a43edb0df92463a498258 (127.0.99.134:42343)
I20240503 01:59:02.597252 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:38939
I20240503 01:59:02.622671 3724 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
W20240503 01:59:02.702929 4382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:02.703187 4382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:02.703230 4382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:02.706517 4382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:02.706609 4382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:02.706688 4382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:02.710520 4382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.4382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:02.711464 4382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:02.712636 4382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:02.721949 4390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.723043 4388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:02.723819 4387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:02.724941 4382 server_base.cc:1034] running on GCE node
I20240503 01:59:02.725426 4382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:02.726061 4382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:02.727216 4382 hybrid_clock.cc:648] HybridClock initialized: now 1714701542727195 us; error 39 us; skew 500 ppm
I20240503 01:59:02.729295 4382 webserver.cc:460] Webserver started at http://127.0.99.135:44141/ using document root <none> and password file <none>
I20240503 01:59:02.729883 4382 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:02.729945 4382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:02.730136 4382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:02.732324 4382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "0fb6a1bb16214e3c9ec6acd1ab111358"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.732923 4382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "0fb6a1bb16214e3c9ec6acd1ab111358"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.736660 4382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:02.739179 4396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.740329 4382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:02.740475 4382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0fb6a1bb16214e3c9ec6acd1ab111358"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:02.740579 4382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:02.754741 4382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:02.755470 4382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:02.755677 4382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:02.756348 4382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:02.757403 4382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:02.757458 4382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.757524 4382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:02.757555 4382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:02.767019 4382 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:42189
I20240503 01:59:02.767048 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:42189 every 8 connection(s)
I20240503 01:59:02.768103 4382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:02.776793 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 4382
I20240503 01:59:02.776945 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:02.777849 4510 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:02.778124 4510 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:02.778828 4510 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:02.779932 3535 ts_manager.cc:194] Registered new tserver with Master: 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189)
I20240503 01:59:02.780505 3535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:57083
I20240503 01:59:02.791574 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:59:02.801877 3535 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:59:02.808744 3855 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:02.819614 4051 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e73129def488bbf9ed086e424968e (DEFAULT_TABLE table=TestTable [id=ffaa0fd37f784a1791ea2e12533d2bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:02.819614 4182 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e73129def488bbf9ed086e424968e (DEFAULT_TABLE table=TestTable [id=ffaa0fd37f784a1791ea2e12533d2bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:02.820593 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e73129def488bbf9ed086e424968e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:02.820593 4051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e73129def488bbf9ed086e424968e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:02.825706 4526 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap starting.
I20240503 01:59:02.825714 4527 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap starting.
I20240503 01:59:02.827792 4527 tablet_bootstrap.cc:654] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:02.827792 4526 tablet_bootstrap.cc:654] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:02.828482 4527 log.cc:824] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:02.828482 4526 log.cc:824] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:02.830144 4526 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: No bootstrap required, opened a new log
I20240503 01:59:02.830143 4527 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: No bootstrap required, opened a new log
I20240503 01:59:02.830322 4527 ts_tablet_manager.cc:1367] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:02.830325 4526 ts_tablet_manager.cc:1367] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:02.832971 4526 raft_consensus.cc:348] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.833184 4526 raft_consensus.cc:374] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:02.833250 4526 raft_consensus.cc:729] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Initialized, Role: FOLLOWER
I20240503 01:59:02.833362 4527 raft_consensus.cc:348] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.833567 4527 raft_consensus.cc:374] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:02.833629 4527 raft_consensus.cc:729] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Initialized, Role: FOLLOWER
I20240503 01:59:02.833707 4526 consensus_queue.cc:260] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.834028 4527 consensus_queue.cc:260] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.834470 4117 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:02.834619 4526 ts_tablet_manager.cc:1398] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:02.834751 4527 ts_tablet_manager.cc:1398] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:02.835104 4248 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
W20240503 01:59:02.894048 4249 tablet.cc:2369] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:02.945699 4118 tablet.cc:2369] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:02.965867 4530 raft_consensus.cc:482] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:02.966092 4530 raft_consensus.cc:504] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.967204 4530 leader_election.cc:288] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:02.971295 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e73129def488bbf9ed086e424968e" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7" is_pre_election: true
I20240503 01:59:02.971669 4202 raft_consensus.cc:2454] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 0.
I20240503 01:59:02.972115 4007 leader_election.cc:302] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b5c836ac65b442e808f46fb2d267d2c, c6f646ec3edd4e298f16122028bb0ef7; no voters:
I20240503 01:59:02.972389 4530 raft_consensus.cc:2781] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:02.972476 4530 raft_consensus.cc:482] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:02.972538 4530 raft_consensus.cc:3037] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:02.973446 4530 raft_consensus.cc:504] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.973757 4530 leader_election.cc:288] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 1 election: Requested vote from peers c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:02.974191 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e73129def488bbf9ed086e424968e" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7"
I20240503 01:59:02.974306 4202 raft_consensus.cc:3037] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:02.975288 4202 raft_consensus.cc:2454] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 1.
I20240503 01:59:02.975651 4007 leader_election.cc:302] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b5c836ac65b442e808f46fb2d267d2c, c6f646ec3edd4e298f16122028bb0ef7; no voters:
I20240503 01:59:02.975839 4530 raft_consensus.cc:2781] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:02.976117 4530 raft_consensus.cc:686] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEADER]: Becoming Leader. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Running, Role: LEADER
I20240503 01:59:02.976469 4530 consensus_queue.cc:237] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:02.979718 3536 catalog_manager.cc:5484] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7b5c836ac65b442e808f46fb2d267d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:03.005489 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:59:03.006901 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2820cbd55ee1470880e35a8ab671563b to finish bootstrapping
I20240503 01:59:03.012845 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f929df19fdfc4df1986e990c60056733 to finish bootstrapping
I20240503 01:59:03.014034 3986 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:03.018841 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c8595f9ae8f644d6901791810d7b970e to finish bootstrapping
I20240503 01:59:03.024391 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7b5c836ac65b442e808f46fb2d267d2c to finish bootstrapping
I20240503 01:59:03.029126 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c6f646ec3edd4e298f16122028bb0ef7 to finish bootstrapping
I20240503 01:59:03.034106 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7a367744629a43edb0df92463a498258 to finish bootstrapping
I20240503 01:59:03.040194 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0fb6a1bb16214e3c9ec6acd1ab111358 to finish bootstrapping
I20240503 01:59:03.046018 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3989
I20240503 01:59:03.052474 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 4120
I20240503 01:59:03.058640 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 4251
I20240503 01:59:03.064393 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 4382
I20240503 01:59:06.821642 3536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:06.835140 3657 tablet_service.cc:1467] Processing CreateTablet for tablet 4baa1b72653c4a0ba130008b06bc5897 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.835152 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7cf094bb2546bb800180d3a530f46a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.835409 3789 tablet_service.cc:1467] Processing CreateTablet for tablet 4baa1b72653c4a0ba130008b06bc5897 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.836102 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4baa1b72653c4a0ba130008b06bc5897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.836114 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4baa1b72653c4a0ba130008b06bc5897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.836114 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7cf094bb2546bb800180d3a530f46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.837488 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7cf094bb2546bb800180d3a530f46a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.837836 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7cf094bb2546bb800180d3a530f46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.842130 3919 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6b5cb588234440807eee9875c334c1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.842157 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6b5cb588234440807eee9875c334c1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:06.842422 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5cb588234440807eee9875c334c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.842422 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5cb588234440807eee9875c334c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.842543 4551 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:06.843078 4552 tablet_bootstrap.cc:492] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:06.844780 4551 tablet_bootstrap.cc:654] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:06.844954 4553 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:06.845279 4552 tablet_bootstrap.cc:654] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:06.845623 4551 log.cc:824] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:06.845939 4552 log.cc:824] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:06.846596 4553 tablet_bootstrap.cc:654] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:06.847242 4553 log.cc:824] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:06.847462 4551 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:06.847653 4551 ts_tablet_manager.cc:1367] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:06.848191 4552 tablet_bootstrap.cc:492] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:06.848358 4552 ts_tablet_manager.cc:1367] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:06.848802 4553 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:06.848963 4553 ts_tablet_manager.cc:1367] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 01:59:06.850618 4551 raft_consensus.cc:348] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.850802 4551 raft_consensus.cc:374] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.850878 4551 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.851294 4551 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.852180 4551 ts_tablet_manager.cc:1398] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 01:59:06.852335 4552 raft_consensus.cc:348] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.852478 4551 tablet_bootstrap.cc:492] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:06.852578 4552 raft_consensus.cc:374] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.852666 4552 raft_consensus.cc:729] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.853137 4552 consensus_queue.cc:260] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.853729 4551 tablet_bootstrap.cc:654] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:06.853827 4553 raft_consensus.cc:348] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.854039 4552 ts_tablet_manager.cc:1398] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:59:06.854072 4553 raft_consensus.cc:374] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.854131 4553 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.854611 4553 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.855022 4551 tablet_bootstrap.cc:492] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:06.855106 4551 ts_tablet_manager.cc:1367] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:06.855453 4553 ts_tablet_manager.cc:1398] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240503 01:59:06.855597 4552 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:06.855728 4551 raft_consensus.cc:348] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.855803 4553 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:06.855821 4551 raft_consensus.cc:374] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.855860 4551 raft_consensus.cc:729] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.855983 4551 consensus_queue.cc:260] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.856240 4551 ts_tablet_manager.cc:1398] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:06.857014 4553 tablet_bootstrap.cc:654] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:06.857623 4552 tablet_bootstrap.cc:654] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
W20240503 01:59:06.858191 3725 tablet.cc:2369] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:06.858875 4553 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:06.858999 4553 ts_tablet_manager.cc:1367] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:06.859735 4553 raft_consensus.cc:348] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.859884 4553 raft_consensus.cc:374] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.859977 4553 raft_consensus.cc:729] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.860141 4553 consensus_queue.cc:260] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.860750 4553 ts_tablet_manager.cc:1398] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:06.860939 4552 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:06.861027 4552 ts_tablet_manager.cc:1367] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:06.861502 4552 raft_consensus.cc:348] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.861611 4552 raft_consensus.cc:374] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:06.861654 4552 raft_consensus.cc:729] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.861771 4552 consensus_queue.cc:260] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:06.862071 4552 ts_tablet_manager.cc:1398] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:06.981232 4557 raft_consensus.cc:482] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:06.981446 4557 raft_consensus.cc:504] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.982514 4557 leader_election.cc:288] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:06.986243 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4baa1b72653c4a0ba130008b06bc5897" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:06.986557 3809 raft_consensus.cc:2454] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:06.987011 3611 leader_election.cc:302] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:06.987296 4557 raft_consensus.cc:2781] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:06.987385 4557 raft_consensus.cc:482] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:06.987432 4557 raft_consensus.cc:3037] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:06.988420 4557 raft_consensus.cc:504] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.988737 4557 leader_election.cc:288] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:06.989207 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4baa1b72653c4a0ba130008b06bc5897" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:06.989338 3809 raft_consensus.cc:3037] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:06.990250 3809 raft_consensus.cc:2454] T 4baa1b72653c4a0ba130008b06bc5897 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:06.990573 3611 leader_election.cc:302] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:06.990756 4557 raft_consensus.cc:2781] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:06.990967 4557 raft_consensus.cc:686] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:06.991262 4557 consensus_queue.cc:237] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:06.994230 3535 catalog_manager.cc:5484] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:07.003196 3987 tablet.cc:2369] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:07.039963 4560 raft_consensus.cc:482] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:07.040176 4560 raft_consensus.cc:504] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.041263 4560 leader_election.cc:288] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:07.044981 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5cb588234440807eee9875c334c1" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:07.045159 3809 raft_consensus.cc:2454] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 0.
I20240503 01:59:07.045579 3873 leader_election.cc:302] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.045859 4560 raft_consensus.cc:2781] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:07.045930 4560 raft_consensus.cc:482] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:07.045959 4560 raft_consensus.cc:3037] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.047127 4560 raft_consensus.cc:504] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.047441 4560 leader_election.cc:288] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:07.047973 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5cb588234440807eee9875c334c1" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:07.048105 3809 raft_consensus.cc:3037] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:07.048838 3856 tablet.cc:2369] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:07.048929 3809 raft_consensus.cc:2454] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 1.
I20240503 01:59:07.049218 3873 leader_election.cc:302] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.049374 4560 raft_consensus.cc:2781] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:07.049602 4560 raft_consensus.cc:686] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:07.050040 4560 consensus_queue.cc:237] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.053143 3535 catalog_manager.cc:5484] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:07.132030 4557 raft_consensus.cc:482] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:07.132215 4557 raft_consensus.cc:504] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.132686 4557 leader_election.cc:288] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.136348 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e" is_pre_election: true
I20240503 01:59:07.136603 3940 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:07.137013 3612 leader_election.cc:302] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:07.137236 4557 raft_consensus.cc:2781] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:07.137306 4557 raft_consensus.cc:482] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:07.137336 4557 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.138119 4557 raft_consensus.cc:504] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.138407 4557 leader_election.cc:288] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.138849 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:07.138975 3940 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.139889 3940 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:07.140249 3612 leader_election.cc:302] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:07.140439 4557 raft_consensus.cc:2781] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:07.140544 4557 raft_consensus.cc:686] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:07.140693 4557 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.142383 3535 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:07.301036 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:07.301293 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:07.304399 4570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:07.386687 4557 consensus_queue.cc:1035] T 4baa1b72653c4a0ba130008b06bc5897 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:07.406834 3940 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:07.407619 4564 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:07.408558 3809 raft_consensus.cc:1261] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:07.409086 4560 consensus_queue.cc:1035] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:07.417820 4605 mvcc.cc:205] Tried to move back new op lower bound from 7023417538169745408 to 7023417537089380352. Current Snapshot: MvccSnapshot[applied={T|T < 7023417538169745408}]
I20240503 01:59:07.427316 4606 mvcc.cc:205] Tried to move back new op lower bound from 7023417538178465792 to 7023417536719380480. Current Snapshot: MvccSnapshot[applied={T|T < 7023417538178465792}]
I20240503 01:59:07.430444 4598 mvcc.cc:205] Tried to move back new op lower bound from 7023417538235162624 to 7023417537089380352. Current Snapshot: MvccSnapshot[applied={T|T < 7023417538169745408}]
INSERT report
rows total: 2000
time total: 107.233 ms
time per row: 0.0536164 ms
I20240503 01:59:07.460153 3529 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:07.465503 3657 tablet_service.cc:1467] Processing CreateTablet for tablet f8d39433e68b4bd7b14b42e850f42ae6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.465569 3789 tablet_service.cc:1467] Processing CreateTablet for tablet f8d39433e68b4bd7b14b42e850f42ae6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.465868 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d39433e68b4bd7b14b42e850f42ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.465848 3658 tablet_service.cc:1467] Processing CreateTablet for tablet cbba240087984f5bac2370d46a09a436 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.466151 3920 tablet_service.cc:1467] Processing CreateTablet for tablet e6c983adafca4912aa70f99c81bb4b4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.465850 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d39433e68b4bd7b14b42e850f42ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.466306 3919 tablet_service.cc:1467] Processing CreateTablet for tablet cbba240087984f5bac2370d46a09a436 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.466423 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c983adafca4912aa70f99c81bb4b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.466521 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbba240087984f5bac2370d46a09a436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.467824 3788 tablet_service.cc:1467] Processing CreateTablet for tablet e6c983adafca4912aa70f99c81bb4b4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.468039 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c983adafca4912aa70f99c81bb4b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.468423 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbba240087984f5bac2370d46a09a436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.468966 4624 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:07.469830 4623 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:07.470139 4624 tablet_bootstrap.cc:654] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.470790 4625 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:07.471026 4623 tablet_bootstrap.cc:654] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.471446 4624 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:07.471520 4624 ts_tablet_manager.cc:1367] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:07.471798 4625 tablet_bootstrap.cc:654] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.472141 4624 raft_consensus.cc:348] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.472239 4624 raft_consensus.cc:374] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.472290 4624 raft_consensus.cc:729] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.472424 4624 consensus_queue.cc:260] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.472608 4623 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:07.472702 4624 ts_tablet_manager.cc:1398] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:07.472712 4623 ts_tablet_manager.cc:1367] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:07.472889 4624 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:07.473145 4623 raft_consensus.cc:348] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.473256 4623 raft_consensus.cc:374] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.473306 4623 raft_consensus.cc:729] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.473445 4623 consensus_queue.cc:260] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.473197 4625 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:07.473585 4625 ts_tablet_manager.cc:1367] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:07.473704 4623 ts_tablet_manager.cc:1398] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:07.473882 4623 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:07.474061 4625 raft_consensus.cc:348] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.474167 4624 tablet_bootstrap.cc:654] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.474169 4625 raft_consensus.cc:374] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.474200 4625 raft_consensus.cc:729] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.474707 4623 tablet_bootstrap.cc:654] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.475749 4625 consensus_queue.cc:260] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.475967 4623 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:07.476056 4623 ts_tablet_manager.cc:1367] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:07.476121 4625 ts_tablet_manager.cc:1398] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:07.476372 4625 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:07.476514 4623 raft_consensus.cc:348] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.476593 4623 raft_consensus.cc:374] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.476621 4623 raft_consensus.cc:729] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.476754 4623 consensus_queue.cc:260] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.477104 4623 ts_tablet_manager.cc:1398] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:07.477295 4625 tablet_bootstrap.cc:654] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.477802 4624 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:07.477869 4624 ts_tablet_manager.cc:1367] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:07.478353 4624 raft_consensus.cc:348] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.478457 4624 raft_consensus.cc:374] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.478498 4624 raft_consensus.cc:729] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.478648 4624 consensus_queue.cc:260] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.479614 4625 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:07.479698 4625 ts_tablet_manager.cc:1367] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:07.479732 4624 ts_tablet_manager.cc:1398] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:07.480064 4625 raft_consensus.cc:348] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.480150 4625 raft_consensus.cc:374] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.480186 4625 raft_consensus.cc:729] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.480331 4625 consensus_queue.cc:260] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.480585 4625 ts_tablet_manager.cc:1398] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:07.489055 4611 raft_consensus.cc:482] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:07.489166 4611 raft_consensus.cc:504] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.489528 4611 leader_election.cc:288] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.489933 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbba240087984f5bac2370d46a09a436" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e" is_pre_election: true
I20240503 01:59:07.490087 3940 raft_consensus.cc:2454] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:07.490363 3612 leader_election.cc:302] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:07.490545 4611 raft_consensus.cc:2781] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:07.490599 4611 raft_consensus.cc:482] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:07.490629 4611 raft_consensus.cc:3037] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.491495 4611 raft_consensus.cc:504] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.491817 4611 leader_election.cc:288] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.492168 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbba240087984f5bac2370d46a09a436" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:07.492272 3940 raft_consensus.cc:3037] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.492987 3940 raft_consensus.cc:2454] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:07.493332 3612 leader_election.cc:302] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:07.493518 4611 raft_consensus.cc:2781] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:07.493592 4611 raft_consensus.cc:686] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:07.493729 4611 consensus_queue.cc:237] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.495276 3529 catalog_manager.cc:5484] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:07.609177 4593 raft_consensus.cc:482] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:07.609333 4593 raft_consensus.cc:504] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.610497 4593 leader_election.cc:288] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.614979 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c983adafca4912aa70f99c81bb4b4f" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e" is_pre_election: true
I20240503 01:59:07.615140 3940 raft_consensus.cc:2454] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 0.
I20240503 01:59:07.615482 3743 leader_election.cc:302] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.615741 4593 raft_consensus.cc:2781] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:07.615799 4593 raft_consensus.cc:482] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:07.615839 4593 raft_consensus.cc:3037] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.616663 4593 raft_consensus.cc:504] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.616956 4593 leader_election.cc:288] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:07.617285 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c983adafca4912aa70f99c81bb4b4f" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:07.617383 3940 raft_consensus.cc:3037] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.618094 3940 raft_consensus.cc:2454] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 1.
I20240503 01:59:07.618352 3743 leader_election.cc:302] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.618526 4593 raft_consensus.cc:2781] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:07.618705 4593 raft_consensus.cc:686] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Becoming Leader. State: Replica: f929df19fdfc4df1986e990c60056733, State: Running, Role: LEADER
I20240503 01:59:07.618974 4593 consensus_queue.cc:237] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.621042 3529 catalog_manager.cc:5484] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 reported cstate change: term changed from 0 to 1, leader changed from <none> to f929df19fdfc4df1986e990c60056733 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:07.636111 4611 raft_consensus.cc:482] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:07.636265 4611 raft_consensus.cc:504] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.636693 4611 leader_election.cc:288] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:07.637236 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:07.637387 3809 raft_consensus.cc:2454] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:07.637666 3611 leader_election.cc:302] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.637830 4611 raft_consensus.cc:2781] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:07.637884 4611 raft_consensus.cc:482] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:07.637910 4611 raft_consensus.cc:3037] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.638692 4611 raft_consensus.cc:504] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.638937 4611 leader_election.cc:288] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:07.639318 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:07.639421 3809 raft_consensus.cc:3037] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:07.640133 3809 raft_consensus.cc:2454] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:07.640396 3611 leader_election.cc:302] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:07.640538 4611 raft_consensus.cc:2781] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:07.640614 4611 raft_consensus.cc:686] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:07.640751 4611 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.642306 3529 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:07.765761 4637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:07.766033 4637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:07.769223 4637 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:07.856204 3809 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:07.856204 3940 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:07.856787 4611 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:07.864854 4557 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:07.873919 3940 raft_consensus.cc:1261] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:07.874363 4593 consensus_queue.cc:1035] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.4527 ms
time per row: 0.0477264 ms
I20240503 01:59:07.906754 3528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:07.912127 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 5dba64ef1acb4293be37a986fc18a105 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.912346 3789 tablet_service.cc:1467] Processing CreateTablet for tablet 5dba64ef1acb4293be37a986fc18a105 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.912467 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba64ef1acb4293be37a986fc18a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.912505 3920 tablet_service.cc:1467] Processing CreateTablet for tablet e504950788d7408a83d4b7299b78518c (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.912624 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba64ef1acb4293be37a986fc18a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.912667 3919 tablet_service.cc:1467] Processing CreateTablet for tablet b1aa056182cf435d8a6f675aa2b42129 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.912757 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e504950788d7408a83d4b7299b78518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.912894 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aa056182cf435d8a6f675aa2b42129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.913408 3788 tablet_service.cc:1467] Processing CreateTablet for tablet b1aa056182cf435d8a6f675aa2b42129 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.913642 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aa056182cf435d8a6f675aa2b42129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.914871 4623 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:07.914952 4625 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:07.915638 4623 tablet_bootstrap.cc:654] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.915808 4624 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:07.915987 4625 tablet_bootstrap.cc:654] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.916872 4624 tablet_bootstrap.cc:654] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.917207 4625 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:07.917294 4625 ts_tablet_manager.cc:1367] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:07.917690 4623 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:07.917765 4623 ts_tablet_manager.cc:1367] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:07.917795 4625 raft_consensus.cc:348] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.917922 4625 raft_consensus.cc:374] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.917965 4625 raft_consensus.cc:729] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.918110 4625 consensus_queue.cc:260] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.918215 4623 raft_consensus.cc:348] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.918315 4623 raft_consensus.cc:374] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.918342 4623 raft_consensus.cc:729] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.918398 4625 ts_tablet_manager.cc:1398] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:07.918440 4623 consensus_queue.cc:260] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.918535 4625 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:07.918663 4623 ts_tablet_manager.cc:1398] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:07.918774 4623 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:07.919373 3658 tablet_service.cc:1467] Processing CreateTablet for tablet e504950788d7408a83d4b7299b78518c (DEFAULT_TABLE table=rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:07.919462 4625 tablet_bootstrap.cc:654] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.919587 4623 tablet_bootstrap.cc:654] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.919512 4624 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:07.919646 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e504950788d7408a83d4b7299b78518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:07.919682 4624 ts_tablet_manager.cc:1367] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 01:59:07.920251 4624 raft_consensus.cc:348] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.920382 4624 raft_consensus.cc:374] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.920413 4624 raft_consensus.cc:729] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.920593 4624 consensus_queue.cc:260] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:07.920971 4624 ts_tablet_manager.cc:1398] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:07.921684 4623 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:07.921759 4623 ts_tablet_manager.cc:1367] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:07.921800 4625 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:07.921916 4625 ts_tablet_manager.cc:1367] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:07.922255 4623 raft_consensus.cc:348] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.922358 4623 raft_consensus.cc:374] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.922392 4623 raft_consensus.cc:729] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.922345 4625 raft_consensus.cc:348] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.922443 4625 raft_consensus.cc:374] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.922480 4625 raft_consensus.cc:729] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.922606 4624 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:07.922605 4625 consensus_queue.cc:260] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:07.922880 4625 ts_tablet_manager.cc:1398] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:07.923638 4624 tablet_bootstrap.cc:654] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:07.923956 4623 consensus_queue.cc:260] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.924808 4624 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:07.924899 4624 ts_tablet_manager.cc:1367] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:07.925767 4623 ts_tablet_manager.cc:1398] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:07.925772 4624 raft_consensus.cc:348] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.925892 4624 raft_consensus.cc:374] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:07.925930 4624 raft_consensus.cc:729] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:07.926070 4624 consensus_queue.cc:260] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:07.926350 4624 ts_tablet_manager.cc:1398] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:08.052610 4557 raft_consensus.cc:482] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.052749 4557 raft_consensus.cc:504] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.053180 4557 leader_election.cc:288] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:08.053570 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba64ef1acb4293be37a986fc18a105" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:08.053694 3809 raft_consensus.cc:2454] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:08.053937 3611 leader_election.cc:302] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.054085 4557 raft_consensus.cc:2781] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.054154 4557 raft_consensus.cc:482] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.054185 4557 raft_consensus.cc:3037] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.055145 4557 raft_consensus.cc:504] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.055430 4557 leader_election.cc:288] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:08.055783 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba64ef1acb4293be37a986fc18a105" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:08.055893 3809 raft_consensus.cc:3037] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.056650 3809 raft_consensus.cc:2454] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:08.056977 3611 leader_election.cc:302] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.057142 4557 raft_consensus.cc:2781] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.057245 4557 raft_consensus.cc:686] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:08.057385 4557 consensus_queue.cc:237] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.058880 3532 catalog_manager.cc:5484] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:08.079893 4557 raft_consensus.cc:482] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.080073 4557 raft_consensus.cc:504] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.080499 4557 leader_election.cc:288] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:08.081012 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504950788d7408a83d4b7299b78518c" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e" is_pre_election: true
I20240503 01:59:08.081172 3940 raft_consensus.cc:2454] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:08.081534 3612 leader_election.cc:302] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:08.081743 4557 raft_consensus.cc:2781] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.081804 4557 raft_consensus.cc:482] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.081868 4557 raft_consensus.cc:3037] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.082727 4557 raft_consensus.cc:504] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.083024 4557 leader_election.cc:288] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:08.083400 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504950788d7408a83d4b7299b78518c" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:08.083562 3940 raft_consensus.cc:3037] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.084374 3940 raft_consensus.cc:2454] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:08.084661 3612 leader_election.cc:302] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:08.084821 4557 raft_consensus.cc:2781] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.084911 4557 raft_consensus.cc:686] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:08.085044 4557 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.086675 3532 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:08.190150 4560 raft_consensus.cc:482] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.190291 4560 raft_consensus.cc:504] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.190697 4560 leader_election.cc:288] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:08.191252 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:08.191406 3809 raft_consensus.cc:2454] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 0.
I20240503 01:59:08.191717 3873 leader_election.cc:302] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.191924 4560 raft_consensus.cc:2781] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.191980 4560 raft_consensus.cc:482] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.192008 4560 raft_consensus.cc:3037] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.192796 4560 raft_consensus.cc:504] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.193113 4560 leader_election.cc:288] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:08.193460 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:08.193574 3809 raft_consensus.cc:3037] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.194315 3809 raft_consensus.cc:2454] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 1.
I20240503 01:59:08.194588 3873 leader_election.cc:302] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.194767 4560 raft_consensus.cc:2781] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.194872 4560 raft_consensus.cc:686] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:08.195010 4560 consensus_queue.cc:237] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.196519 3532 catalog_manager.cc:5484] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:08.315132 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:08.315368 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:08.318297 4677 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:08.410588 3809 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:08.411156 4560 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:08.411929 3940 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:08.412429 3808 raft_consensus.cc:1261] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:08.412515 4557 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:08.413035 4557 consensus_queue.cc:1035] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.712 ms
time per row: 0.051856 ms
I20240503 01:59:08.463189 3534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:08.468380 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 6365fecf9fa04d6eaf6ce9dbe47add84 (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.468520 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 6365fecf9fa04d6eaf6ce9dbe47add84 (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.468753 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6365fecf9fa04d6eaf6ce9dbe47add84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.468816 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6365fecf9fa04d6eaf6ce9dbe47add84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.469048 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 56cd6bd86d2a4391947f7b7193b5e420 (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.469287 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cd6bd86d2a4391947f7b7193b5e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.470651 4725 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:08.471424 4727 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:08.471578 4725 tablet_bootstrap.cc:654] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.471520 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 02a421366d744afd99d7779644e9e41e (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.471624 3919 tablet_service.cc:1467] Processing CreateTablet for tablet 02a421366d744afd99d7779644e9e41e (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.471772 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a421366d744afd99d7779644e9e41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.471815 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a421366d744afd99d7779644e9e41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.472306 4727 tablet_bootstrap.cc:654] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.473276 4727 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:08.473346 4727 ts_tablet_manager.cc:1367] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:08.473619 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 56cd6bd86d2a4391947f7b7193b5e420 (DEFAULT_TABLE table=rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:08.473843 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cd6bd86d2a4391947f7b7193b5e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:08.473831 4727 raft_consensus.cc:348] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.474032 4726 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:08.474025 4727 raft_consensus.cc:374] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.474068 4727 raft_consensus.cc:729] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.474254 4727 consensus_queue.cc:260] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.474598 4727 ts_tablet_manager.cc:1398] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:08.474838 4727 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:08.474890 4726 tablet_bootstrap.cc:654] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.475876 4727 tablet_bootstrap.cc:654] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.475960 4726 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:08.476037 4726 ts_tablet_manager.cc:1367] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:08.476356 4725 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:08.476431 4725 ts_tablet_manager.cc:1367] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:08.476488 4726 raft_consensus.cc:348] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.476574 4726 raft_consensus.cc:374] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.476596 4726 raft_consensus.cc:729] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.476704 4726 consensus_queue.cc:260] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.476863 4725 raft_consensus.cc:348] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.476969 4725 raft_consensus.cc:374] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.476984 4726 ts_tablet_manager.cc:1398] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:08.477002 4725 raft_consensus.cc:729] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.477118 4725 consensus_queue.cc:260] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.477389 4725 ts_tablet_manager.cc:1398] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:08.477406 4726 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:08.477559 4725 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:08.478219 4726 tablet_bootstrap.cc:654] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.478363 4725 tablet_bootstrap.cc:654] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:08.479226 4726 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:08.479296 4726 ts_tablet_manager.cc:1367] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:08.479766 4725 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:08.479728 4726 raft_consensus.cc:348] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.479811 4726 raft_consensus.cc:374] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.479840 4726 raft_consensus.cc:729] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.479842 4725 ts_tablet_manager.cc:1367] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:08.479972 4726 consensus_queue.cc:260] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.480187 4726 ts_tablet_manager.cc:1398] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:08.480268 4725 raft_consensus.cc:348] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.480381 4725 raft_consensus.cc:374] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.480428 4725 raft_consensus.cc:729] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.480546 4725 consensus_queue.cc:260] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.480660 4727 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:08.480749 4727 ts_tablet_manager.cc:1367] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:08.480791 4725 ts_tablet_manager.cc:1398] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:08.481169 4727 raft_consensus.cc:348] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.481268 4727 raft_consensus.cc:374] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:08.481313 4727 raft_consensus.cc:729] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:08.481427 4727 consensus_queue.cc:260] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.481664 4727 ts_tablet_manager.cc:1398] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:08.522984 4593 raft_consensus.cc:482] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.523149 4593 raft_consensus.cc:504] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.523588 4593 leader_election.cc:288] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:08.524159 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a421366d744afd99d7779644e9e41e" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e" is_pre_election: true
I20240503 01:59:08.524304 3939 raft_consensus.cc:2454] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 0.
I20240503 01:59:08.524624 3743 leader_election.cc:302] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.524822 4593 raft_consensus.cc:2781] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.524888 4593 raft_consensus.cc:482] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.524943 4593 raft_consensus.cc:3037] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.525951 4593 raft_consensus.cc:504] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.526218 4593 leader_election.cc:288] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:08.526587 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a421366d744afd99d7779644e9e41e" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:08.526702 3939 raft_consensus.cc:3037] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.527514 3939 raft_consensus.cc:2454] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 1.
I20240503 01:59:08.527824 3743 leader_election.cc:302] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.527992 4593 raft_consensus.cc:2781] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.528194 4593 raft_consensus.cc:686] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Becoming Leader. State: Replica: f929df19fdfc4df1986e990c60056733, State: Running, Role: LEADER
I20240503 01:59:08.528342 4593 consensus_queue.cc:237] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:08.529881 3536 catalog_manager.cc:5484] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 reported cstate change: term changed from 0 to 1, leader changed from <none> to f929df19fdfc4df1986e990c60056733 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:08.619446 4714 raft_consensus.cc:482] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.619635 4714 raft_consensus.cc:504] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.620108 4714 leader_election.cc:288] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:08.623862 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cd6bd86d2a4391947f7b7193b5e420" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b" is_pre_election: true
I20240503 01:59:08.624078 3678 raft_consensus.cc:2454] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 0.
I20240503 01:59:08.624403 3874 leader_election.cc:302] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:08.624614 4714 raft_consensus.cc:2781] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.624680 4714 raft_consensus.cc:482] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.624725 4714 raft_consensus.cc:3037] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.625784 4714 raft_consensus.cc:504] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.626085 4714 leader_election.cc:288] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:08.626448 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cd6bd86d2a4391947f7b7193b5e420" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:08.626576 3678 raft_consensus.cc:3037] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.627583 3678 raft_consensus.cc:2454] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 1.
I20240503 01:59:08.627862 3874 leader_election.cc:302] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:08.628041 4714 raft_consensus.cc:2781] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.628135 4714 raft_consensus.cc:686] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:08.628275 4714 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:08.629849 3536 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:08.785004 4593 raft_consensus.cc:482] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:08.785171 4593 raft_consensus.cc:504] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.785579 4593 leader_election.cc:288] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:08.788803 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b" is_pre_election: true
I20240503 01:59:08.788964 3678 raft_consensus.cc:2454] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 0.
I20240503 01:59:08.789223 3743 leader_election.cc:302] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.789395 4593 raft_consensus.cc:2781] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:08.789451 4593 raft_consensus.cc:482] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:08.789516 4593 raft_consensus.cc:3037] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.790297 4593 raft_consensus.cc:504] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.790572 4593 leader_election.cc:288] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:08.790985 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:08.791092 3678 raft_consensus.cc:3037] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:08.791883 3678 raft_consensus.cc:2454] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 1.
I20240503 01:59:08.792161 3743 leader_election.cc:302] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:08.792321 4593 raft_consensus.cc:2781] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:08.792407 4593 raft_consensus.cc:686] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Becoming Leader. State: Replica: f929df19fdfc4df1986e990c60056733, State: Running, Role: LEADER
I20240503 01:59:08.792544 4593 consensus_queue.cc:237] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:08.794356 3536 catalog_manager.cc:5484] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 reported cstate change: term changed from 0 to 1, leader changed from <none> to f929df19fdfc4df1986e990c60056733 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:08.948918 4740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:08.949172 4740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:08.952441 4740 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:09.042510 3678 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:09.042624 3939 raft_consensus.cc:1261] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:09.043394 4734 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:09.043735 4734 consensus_queue.cc:1035] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:09.045032 3677 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:09.045595 4714 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 94.5076 ms
time per row: 0.0472538 ms
I20240503 01:59:09.089995 3534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:09.095607 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2ab9fbe93a44adbc34f306e8d61c27 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.095675 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2ab9fbe93a44adbc34f306e8d61c27 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.095974 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2ab9fbe93a44adbc34f306e8d61c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.095988 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2ab9fbe93a44adbc34f306e8d61c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.096244 3657 tablet_service.cc:1467] Processing CreateTablet for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.096237 3919 tablet_service.cc:1467] Processing CreateTablet for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.096475 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.096506 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.098434 4790 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:09.098742 4792 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:09.098685 3788 tablet_service.cc:1467] Processing CreateTablet for tablet 287bd1cfeb1e4e709d40fb3368aaca77 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.098918 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287bd1cfeb1e4e709d40fb3368aaca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.098912 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 287bd1cfeb1e4e709d40fb3368aaca77 (DEFAULT_TABLE table=rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:09.099141 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287bd1cfeb1e4e709d40fb3368aaca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:09.099315 4790 tablet_bootstrap.cc:654] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.099685 4792 tablet_bootstrap.cc:654] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.099952 4791 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:09.100854 4791 tablet_bootstrap.cc:654] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.101215 4790 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:09.101300 4790 ts_tablet_manager.cc:1367] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:09.101794 4790 raft_consensus.cc:348] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.101912 4790 raft_consensus.cc:374] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.101945 4790 raft_consensus.cc:729] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.102100 4790 consensus_queue.cc:260] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.102438 4792 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:09.102489 4790 ts_tablet_manager.cc:1398] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:09.102509 4791 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:09.102521 4792 ts_tablet_manager.cc:1367] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:09.102586 4791 ts_tablet_manager.cc:1367] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:09.102634 4790 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Bootstrap starting.
I20240503 01:59:09.102943 4792 raft_consensus.cc:348] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.103052 4792 raft_consensus.cc:374] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.103005 4791 raft_consensus.cc:348] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.103114 4792 raft_consensus.cc:729] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.103128 4791 raft_consensus.cc:374] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.103166 4791 raft_consensus.cc:729] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.103248 4792 consensus_queue.cc:260] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.103276 4791 consensus_queue.cc:260] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.103504 4792 ts_tablet_manager.cc:1398] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:09.103536 4791 ts_tablet_manager.cc:1398] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:09.103688 4792 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Bootstrap starting.
I20240503 01:59:09.103778 4790 tablet_bootstrap.cc:654] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.104095 4791 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Bootstrap starting.
I20240503 01:59:09.104799 4790 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: No bootstrap required, opened a new log
I20240503 01:59:09.104822 4792 tablet_bootstrap.cc:654] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.104878 4790 ts_tablet_manager.cc:1367] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:09.104916 4791 tablet_bootstrap.cc:654] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:09.105296 4790 raft_consensus.cc:348] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.105379 4790 raft_consensus.cc:374] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.105413 4790 raft_consensus.cc:729] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.105527 4790 consensus_queue.cc:260] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.105780 4790 ts_tablet_manager.cc:1398] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:09.105986 4791 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: No bootstrap required, opened a new log
I20240503 01:59:09.106067 4791 ts_tablet_manager.cc:1367] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:09.106452 4791 raft_consensus.cc:348] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.106570 4791 raft_consensus.cc:374] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.106606 4791 raft_consensus.cc:729] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.106732 4791 consensus_queue.cc:260] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.106894 4792 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: No bootstrap required, opened a new log
I20240503 01:59:09.106971 4792 ts_tablet_manager.cc:1367] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:09.106981 4791 ts_tablet_manager.cc:1398] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:09.107342 4792 raft_consensus.cc:348] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.107427 4792 raft_consensus.cc:374] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:09.107470 4792 raft_consensus.cc:729] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.107641 4792 consensus_queue.cc:260] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.107908 4792 ts_tablet_manager.cc:1398] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:09.155125 4557 raft_consensus.cc:482] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:09.155279 4557 raft_consensus.cc:504] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.155685 4557 leader_election.cc:288] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:09.156167 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2ab9fbe93a44adbc34f306e8d61c27" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:09.156306 3809 raft_consensus.cc:2454] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 0.
I20240503 01:59:09.156620 3611 leader_election.cc:302] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:09.156795 4557 raft_consensus.cc:2781] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:09.156872 4557 raft_consensus.cc:482] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:09.156925 4557 raft_consensus.cc:3037] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.157778 4557 raft_consensus.cc:504] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.158082 4557 leader_election.cc:288] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:09.158473 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2ab9fbe93a44adbc34f306e8d61c27" candidate_uuid: "2820cbd55ee1470880e35a8ab671563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:09.158599 3809 raft_consensus.cc:3037] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.159363 3809 raft_consensus.cc:2454] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2820cbd55ee1470880e35a8ab671563b in term 1.
I20240503 01:59:09.159685 3611 leader_election.cc:302] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:09.159866 4557 raft_consensus.cc:2781] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:09.159955 4557 raft_consensus.cc:686] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Leader. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:09.160087 4557 consensus_queue.cc:237] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } }
I20240503 01:59:09.161612 3536 catalog_manager.cc:5484] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2820cbd55ee1470880e35a8ab671563b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:09.225870 4784 raft_consensus.cc:482] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:09.226007 4784 raft_consensus.cc:504] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.226390 4784 leader_election.cc:288] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:09.226846 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fef22e6eb4ecebb2ba7c4d7a5b0c5" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b" is_pre_election: true
I20240503 01:59:09.226979 3678 raft_consensus.cc:2454] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 0.
I20240503 01:59:09.227252 3874 leader_election.cc:302] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:09.227432 4784 raft_consensus.cc:2781] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:09.227485 4784 raft_consensus.cc:482] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:09.227528 4784 raft_consensus.cc:3037] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.228367 4784 raft_consensus.cc:504] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.228650 4784 leader_election.cc:288] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:09.229039 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fef22e6eb4ecebb2ba7c4d7a5b0c5" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:09.229169 3678 raft_consensus.cc:3037] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.229900 3678 raft_consensus.cc:2454] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 1.
I20240503 01:59:09.230180 3874 leader_election.cc:302] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:09.230337 4784 raft_consensus.cc:2781] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:09.230427 4784 raft_consensus.cc:686] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:09.230574 4784 consensus_queue.cc:237] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } }
I20240503 01:59:09.232177 3536 catalog_manager.cc:5484] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:09.242367 4784 raft_consensus.cc:482] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:09.242487 4784 raft_consensus.cc:504] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.242774 4784 leader_election.cc:288] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:09.243266 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733" is_pre_election: true
I20240503 01:59:09.243393 3809 raft_consensus.cc:2454] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 0.
I20240503 01:59:09.243723 3873 leader_election.cc:302] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:09.243897 4784 raft_consensus.cc:2781] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:09.243959 4784 raft_consensus.cc:482] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:09.244005 4784 raft_consensus.cc:3037] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.244855 4784 raft_consensus.cc:504] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.245123 4784 leader_election.cc:288] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:09.245523 3809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:09.245631 3809 raft_consensus.cc:3037] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:09.246361 3809 raft_consensus.cc:2454] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 1.
I20240503 01:59:09.246642 3873 leader_election.cc:302] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:09.246791 4784 raft_consensus.cc:2781] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:09.246881 4784 raft_consensus.cc:686] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:09.247014 4784 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } }
I20240503 01:59:09.248891 3536 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 0 to 1, leader changed from <none> to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:09.364279 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:09.364540 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:09.367471 4802 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:09.449347 3809 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:09.449366 3678 raft_consensus.cc:1261] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:09.450104 4714 consensus_queue.cc:1035] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:09.450448 4714 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:09.456301 3808 raft_consensus.cc:1261] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:09.463802 4834 consensus_queue.cc:1035] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.017 ms
time per row: 0.0440085 ms
I20240503 01:59:09.498493 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:35901
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:33221
--webserver_port=35363
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:09.610513 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:09.610823 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:09.610884 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:09.615367 4837 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:09.615469 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:09.615602 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:09.620858 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:35901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=35363
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.4837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:09.622151 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:09.623561 4837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:09.633498 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:09.634240 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:09.635231 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:09.636168 4837 server_base.cc:1034] running on GCE node
I20240503 01:59:09.636607 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:09.637285 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:09.638437 4837 hybrid_clock.cc:648] HybridClock initialized: now 1714701549638425 us; error 31 us; skew 500 ppm
I20240503 01:59:09.640300 4837 webserver.cc:460] Webserver started at http://127.0.99.132:35363/ using document root <none> and password file <none>
I20240503 01:59:09.640874 4837 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:09.640955 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:09.644727 4837 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20240503 01:59:09.646847 4851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:09.647876 4837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:09.647982 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7b5c836ac65b442e808f46fb2d267d2c"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:09.648406 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:09.665869 4837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:09.666579 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:09.666764 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:09.667344 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:09.668625 4858 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 01:59:09.671712 4837 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 01:59:09.671777 4837 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 01:59:09.671875 4837 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 01:59:09.673291 4837 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 01:59:09.673331 4837 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:09.673552 4858 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap starting.
I20240503 01:59:09.683921 4837 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35901
I20240503 01:59:09.683997 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35901 every 8 connection(s)
I20240503 01:59:09.685032 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:09.685402 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 4837
I20240503 01:59:09.685936 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:41793
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:33221
--webserver_port=36207
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:09.687721 4858 log.cc:824] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:09.689860 4858 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:09.690258 4858 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap complete.
I20240503 01:59:09.691236 4858 ts_tablet_manager.cc:1367] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240503 01:59:09.693732 4966 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:09.693979 4966 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:09.694567 4966 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:09.695425 4858 raft_consensus.cc:348] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:09.696179 4858 raft_consensus.cc:729] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.696439 3533 ts_manager.cc:194] Re-registered known tserver with Master: 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901)
I20240503 01:59:09.696746 4858 consensus_queue.cc:260] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:09.697549 3533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:54233
I20240503 01:59:09.697839 4858 ts_tablet_manager.cc:1398] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 01:59:09.699594 4966 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
W20240503 01:59:09.808590 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:09.808867 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:09.808950 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:09.812132 4969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:09.812227 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:09.812343 4969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:09.816123 4969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:41793
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=36207
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.4969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:09.817183 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:09.818442 4969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:09.827920 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:09.828929 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:09.829718 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:09.830927 4969 server_base.cc:1034] running on GCE node
I20240503 01:59:09.831440 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:09.832288 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:09.833451 4969 hybrid_clock.cc:648] HybridClock initialized: now 1714701549833438 us; error 28 us; skew 500 ppm
I20240503 01:59:09.835530 4969 webserver.cc:460] Webserver started at http://127.0.99.133:36207/ using document root <none> and password file <none>
I20240503 01:59:09.836159 4969 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:09.836233 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:09.839722 4969 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20240503 01:59:09.841810 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:09.842818 4969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:09.842934 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c6f646ec3edd4e298f16122028bb0ef7"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:09.843387 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:09.858692 4969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:09.859372 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:09.859580 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:09.860148 4969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:09.861279 4992 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 01:59:09.864476 4969 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 01:59:09.864540 4969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 01:59:09.864603 4969 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 01:59:09.866057 4969 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 01:59:09.866099 4969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:09.866370 4992 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap starting.
I20240503 01:59:09.875885 4969 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41793
I20240503 01:59:09.875908 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41793 every 8 connection(s)
I20240503 01:59:09.877117 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:09.880635 4992 log.cc:824] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:09.881987 4992 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:09.882398 4992 tablet_bootstrap.cc:492] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap complete.
I20240503 01:59:09.883592 4992 ts_tablet_manager.cc:1367] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.010s
I20240503 01:59:09.886265 5100 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:09.886433 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 4969
I20240503 01:59:09.886508 5100 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:09.886854 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:42343
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:33221
--webserver_port=46193
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:09.886667 4992 raft_consensus.cc:348] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:09.887005 4992 raft_consensus.cc:729] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Initialized, Role: FOLLOWER
I20240503 01:59:09.887158 5100 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:09.887465 4992 consensus_queue.cc:260] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:09.888375 4992 ts_tablet_manager.cc:1398] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 01:59:09.889114 3533 ts_manager.cc:194] Re-registered known tserver with Master: c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:09.889832 3533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45113
I20240503 01:59:09.892356 5100 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
W20240503 01:59:09.936906 4967 tablet.cc:2369] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:10.003814 5105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:10.004088 5105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:10.004168 5105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:10.007298 5105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:10.007400 5105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:10.007503 5105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:10.011340 5105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:42343
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=46193
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.5105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:10.012370 5105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:10.013644 5105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:10.023108 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:10.023808 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:10.025013 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:10.025863 5105 server_base.cc:1034] running on GCE node
I20240503 01:59:10.026253 5105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:10.026819 5105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:10.027993 5105 hybrid_clock.cc:648] HybridClock initialized: now 1714701550027974 us; error 42 us; skew 500 ppm
I20240503 01:59:10.030087 5105 webserver.cc:460] Webserver started at http://127.0.99.134:46193/ using document root <none> and password file <none>
I20240503 01:59:10.030637 5105 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:10.030715 5105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:10.034621 5105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:10.036648 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.037685 5105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:10.037791 5105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7a367744629a43edb0df92463a498258"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:10.038219 5105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:10.057511 5105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:10.058207 5105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:10.058370 5105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:10.058938 5105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:10.059846 5105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:10.059909 5105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.059967 5105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:10.059993 5105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.068579 5105 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42343
I20240503 01:59:10.068630 5232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42343 every 8 connection(s)
I20240503 01:59:10.069686 5105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:10.073192 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 5105
I20240503 01:59:10.073621 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:42189
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:33221
--webserver_port=44141
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:10.079035 5233 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:10.079262 5233 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:10.079897 5233 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:10.080969 3533 ts_manager.cc:194] Re-registered known tserver with Master: 7a367744629a43edb0df92463a498258 (127.0.99.134:42343)
I20240503 01:59:10.081514 3533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39165
W20240503 01:59:10.128171 5101 tablet.cc:2369] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:10.190238 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:10.190496 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:10.190572 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:10.193893 5236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:10.193975 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:10.194089 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:10.198204 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:42189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=44141
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.5236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:10.199198 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:10.200376 5236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:10.209185 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:10.209826 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:10.210966 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:10.211910 5236 server_base.cc:1034] running on GCE node
I20240503 01:59:10.212357 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:10.212990 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:10.214156 5236 hybrid_clock.cc:648] HybridClock initialized: now 1714701550214142 us; error 29 us; skew 500 ppm
I20240503 01:59:10.216065 5236 webserver.cc:460] Webserver started at http://127.0.99.135:44141/ using document root <none> and password file <none>
I20240503 01:59:10.216631 5236 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:10.216732 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:10.220193 5236 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:10.222048 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.223096 5236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:10.223237 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0fb6a1bb16214e3c9ec6acd1ab111358"
format_stamp: "Formatted at 2024-05-03 01:59:02 on dist-test-slave-pvfz"
I20240503 01:59:10.223733 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:10.239080 5236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:10.239863 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:10.240025 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:10.240707 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:10.241948 5236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:10.242012 5236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.242126 5236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:10.242182 5236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:10.254092 5236 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:42189
I20240503 01:59:10.254124 5363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:42189 every 8 connection(s)
I20240503 01:59:10.255816 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:10.264828 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 5236
I20240503 01:59:10.265012 398 test_util.cc:268] Using random seed: 281846451
I20240503 01:59:10.269739 5364 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:10.270009 5364 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:10.270720 5364 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:10.271764 3536 ts_manager.cc:194] Re-registered known tserver with Master: 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189)
I20240503 01:59:10.272575 3536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:60111
I20240503 01:59:10.277240 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 01:59:10.277482 398 test_util.cc:268] Using random seed: 281858920
I20240503 01:59:10.288232 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 01:59:10.288414 398 test_util.cc:268] Using random seed: 281869851
I20240503 01:59:10.302421 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 01:59:10.302575 398 test_util.cc:268] Using random seed: 281884014
I20240503 01:59:10.317266 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 01:59:10.317446 398 test_util.cc:268] Using random seed: 281898885
I20240503 01:59:10.330628 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 01:59:10.772722 3657 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:10.773146 5024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.782339 4900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.787844 5167 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.793849 5298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:10.812825 3788 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.874514 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:11.083810 5233 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:11.169855 5167 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:11.169855 3657 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:11.171159 5298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:11.176823 3788 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.178195 5024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:11.179294 4900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:11.216328 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:11.275341 5364 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:11.290777 3940 consensus_queue.cc:237] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:11.293397 3807 raft_consensus.cc:1261] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
W20240503 01:59:11.301826 3875 consensus_peers.cc:485] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e -> Peer 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): Couldn't send request to peer 0fb6a1bb16214e3c9ec6acd1ab111358. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c6b5cb588234440807eee9875c334c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.305356 4836 consensus_queue.cc:1035] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 01:59:11.309006 5551 raft_consensus.cc:482] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.310236 5551 raft_consensus.cc:504] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:11.311997 5551 leader_election.cc:288] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901)
I20240503 01:59:11.311995 5447 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.314618 3809 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.317086 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e73129def488bbf9ed086e424968e" candidate_uuid: "c6f646ec3edd4e298f16122028bb0ef7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5c836ac65b442e808f46fb2d267d2c" is_pre_election: true
I20240503 01:59:11.317414 4919 raft_consensus.cc:2396] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6f646ec3edd4e298f16122028bb0ef7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 01:59:11.318246 4986 leader_election.cc:302] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c6f646ec3edd4e298f16122028bb0ef7; no voters: 7b5c836ac65b442e808f46fb2d267d2c
I20240503 01:59:11.318878 5551 raft_consensus.cc:2726] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:11.318926 3531 catalog_manager.cc:5484] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.359098 3940 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:11.363263 3806 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240503 01:59:11.369959 3875 consensus_peers.cc:485] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e -> Peer 7a367744629a43edb0df92463a498258 (127.0.99.134:42343): Couldn't send request to peer 7a367744629a43edb0df92463a498258. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 287bd1cfeb1e4e709d40fb3368aaca77. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.370158 5433 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 01:59:11.376582 5433 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:11.376821 3807 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:11.380026 3531 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index -1 to 29, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.421492 3677 consensus_queue.cc:237] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:11.434032 3940 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:11.434688 5384 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
W20240503 01:59:11.437024 3614 consensus_peers.cc:485] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b -> Peer c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793): Couldn't send request to peer c6f646ec3edd4e298f16122028bb0ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbba240087984f5bac2370d46a09a436. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.438215 5466 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index -1 to 54, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.438728 3938 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index -1 to 54, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.442091 3534 catalog_manager.cc:5484] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index -1 to 54, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.445909 5568 raft_consensus.cc:482] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.446146 5568 raft_consensus.cc:504] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:11.450039 5568 leader_election.cc:288] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:11.466827 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e73129def488bbf9ed086e424968e" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7" is_pre_election: true
I20240503 01:59:11.467130 5054 raft_consensus.cc:2454] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 1.
I20240503 01:59:11.468001 4855 leader_election.cc:302] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b5c836ac65b442e808f46fb2d267d2c, c6f646ec3edd4e298f16122028bb0ef7; no voters:
I20240503 01:59:11.469352 5568 raft_consensus.cc:2781] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:11.469523 5568 raft_consensus.cc:482] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.469640 5568 raft_consensus.cc:3037] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:11.478456 5568 raft_consensus.cc:504] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:11.479401 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e73129def488bbf9ed086e424968e" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7"
I20240503 01:59:11.479542 5054 raft_consensus.cc:3037] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:11.482019 5054 raft_consensus.cc:2454] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 2.
I20240503 01:59:11.482856 4855 leader_election.cc:302] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b5c836ac65b442e808f46fb2d267d2c, c6f646ec3edd4e298f16122028bb0ef7; no voters:
I20240503 01:59:11.483251 5575 raft_consensus.cc:2781] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:11.483451 5575 raft_consensus.cc:686] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [term 2 LEADER]: Becoming Leader. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Running, Role: LEADER
I20240503 01:59:11.483891 5575 consensus_queue.cc:237] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } }
I20240503 01:59:11.485484 5568 leader_election.cc:288] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 election: Requested vote from peers c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:11.488574 3534 catalog_manager.cc:5484] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "7b5c836ac65b442e808f46fb2d267d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:11.499531 3806 consensus_queue.cc:237] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:11.501127 3678 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 01:59:11.501561 5456 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 01:59:11.503319 5456 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
I20240503 01:59:11.506978 3534 catalog_manager.cc:5484] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index -1 to 55, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.509589 3678 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
W20240503 01:59:11.537336 3742 consensus_peers.cc:485] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 -> Peer 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901): Couldn't send request to peer 7b5c836ac65b442e808f46fb2d267d2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6365fecf9fa04d6eaf6ce9dbe47add84. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.553478 3808 consensus_queue.cc:237] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:11.556684 3938 raft_consensus.cc:1261] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20240503 01:59:11.557217 5533 consensus_queue.cc:1035] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 01:59:11.561718 5423 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
W20240503 01:59:11.564894 3744 consensus_peers.cc:485] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 -> Peer 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): Couldn't send request to peer 0fb6a1bb16214e3c9ec6acd1ab111358. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c983adafca4912aa70f99c81bb4b4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.565511 3938 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.567045 3534 catalog_manager.cc:5484] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index -1 to 40, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.590512 3677 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:11.592053 3806 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 01:59:11.593036 5515 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20240503 01:59:11.595467 5515 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
W20240503 01:59:11.596686 3613 consensus_peers.cc:485] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b -> Peer 7a367744629a43edb0df92463a498258 (127.0.99.134:42343): Couldn't send request to peer 7a367744629a43edb0df92463a498258. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8d39433e68b4bd7b14b42e850f42ae6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.598596 3534 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index -1 to 42, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.601083 3808 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:11.686203 3678 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
W20240503 01:59:11.694129 3611 consensus_peers.cc:485] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b -> Peer 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901): Couldn't send request to peer 7b5c836ac65b442e808f46fb2d267d2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a7cf094bb2546bb800180d3a530f46a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.694350 3939 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 01:59:11.696686 5594 ts_tablet_manager.cc:902] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:11.698796 5406 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 01:59:11.702389 5594 tablet_copy_client.cc:308] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:11.702560 4834 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
I20240503 01:59:11.706612 3939 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
I20240503 01:59:11.707798 3534 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index -1 to 47, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.719651 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet 1c6b5cb588234440807eee9875c334c1 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 ({username='slave'} at 127.0.99.135:48497)
I20240503 01:59:11.719832 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet 1c6b5cb588234440807eee9875c334c1 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 at {username='slave'} at 127.0.99.135:48497: session id = 0fb6a1bb16214e3c9ec6acd1ab111358-1c6b5cb588234440807eee9875c334c1
I20240503 01:59:11.721319 3960 tablet_copy_source_session.cc:205] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:11.732636 3678 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:11.733880 3940 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 01:59:11.734273 5466 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
W20240503 01:59:11.734678 3614 consensus_peers.cc:485] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b -> Peer c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793): Couldn't send request to peer c6f646ec3edd4e298f16122028bb0ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e504950788d7408a83d4b7299b78518c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.737668 5516 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.738626 3940 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.740711 3531 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index -1 to 62, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.752740 5594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5cb588234440807eee9875c334c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.760659 5594 tablet_copy_client.cc:783] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:11.760860 5594 tablet_copy_client.cc:647] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:11.768049 3940 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:11.768893 5594 tablet_copy_client.cc:523] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:11.769361 3677 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
W20240503 01:59:11.769542 3875 consensus_peers.cc:485] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e -> Peer 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): Couldn't send request to peer 0fb6a1bb16214e3c9ec6acd1ab111358. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56cd6bd86d2a4391947f7b7193b5e420. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.769780 4836 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 01:59:11.771276 5594 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap starting.
I20240503 01:59:11.771486 5433 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.775808 3677 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.778215 5610 ts_tablet_manager.cc:902] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:11.779132 5610 tablet_copy_client.cc:308] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:11.788913 5594 log.cc:824] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.791635 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet 287bd1cfeb1e4e709d40fb3368aaca77 from peer 7a367744629a43edb0df92463a498258 ({username='slave'} at 127.0.99.134:46137)
I20240503 01:59:11.791749 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet 287bd1cfeb1e4e709d40fb3368aaca77 from peer 7a367744629a43edb0df92463a498258 at {username='slave'} at 127.0.99.134:46137: session id = 7a367744629a43edb0df92463a498258-287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:11.792699 3960 tablet_copy_source_session.cc:205] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:11.796883 5610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287bd1cfeb1e4e709d40fb3368aaca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.806758 3534 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index -1 to 56, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:11.841295 3678 consensus_queue.cc:237] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:11.842248 5610 tablet_copy_client.cc:783] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:11.842536 5610 tablet_copy_client.cc:647] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 1 WAL segments...
W20240503 01:59:11.844728 3613 consensus_peers.cc:485] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b -> Peer 7a367744629a43edb0df92463a498258 (127.0.99.134:42343): Couldn't send request to peer 7a367744629a43edb0df92463a498258. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dba64ef1acb4293be37a986fc18a105. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.845955 5610 tablet_copy_client.cc:523] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:11.848573 3807 raft_consensus.cc:1261] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240503 01:59:11.854260 5515 consensus_queue.cc:1035] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:11.862067 5406 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:11.869189 3534 catalog_manager.cc:5484] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index -1 to 58, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.869668 5610 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Bootstrap starting.
I20240503 01:59:11.873941 3809 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:11.876958 5594 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=727 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:11.877357 5594 tablet_bootstrap.cc:492] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap complete.
I20240503 01:59:11.877733 5594 ts_tablet_manager.cc:1367] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent bootstrapping tablet: real 0.107s user 0.066s sys 0.007s
I20240503 01:59:11.880271 5594 raft_consensus.cc:348] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:11.880765 5594 raft_consensus.cc:729] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Initialized, Role: LEARNER
I20240503 01:59:11.881714 5594 consensus_queue.cc:260] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:11.881953 5610 log.cc:824] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.882664 5594 ts_tablet_manager.cc:1398] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:11.884279 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-1c6b5cb588234440807eee9875c334c1 received from {username='slave'} at 127.0.99.135:48497
I20240503 01:59:11.884371 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-1c6b5cb588234440807eee9875c334c1 on tablet 1c6b5cb588234440807eee9875c334c1 with peer 0fb6a1bb16214e3c9ec6acd1ab111358
I20240503 01:59:11.894773 3939 consensus_queue.cc:237] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:11.896529 3678 raft_consensus.cc:1261] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240503 01:59:11.896936 5447 consensus_queue.cc:1035] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20240503 01:59:11.898703 4836 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
I20240503 01:59:11.899190 3678 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } } }
W20240503 01:59:11.908650 3873 consensus_peers.cc:485] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e -> Peer 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901): Couldn't send request to peer 7b5c836ac65b442e808f46fb2d267d2c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67fef22e6eb4ecebb2ba7c4d7a5b0c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:11.938983 5630 ts_tablet_manager.cc:902] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:11.939860 5630 tablet_copy_client.cc:308] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:11.944604 3698 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet cbba240087984f5bac2370d46a09a436 from peer c6f646ec3edd4e298f16122028bb0ef7 ({username='slave'} at 127.0.99.133:51435)
I20240503 01:59:11.944787 3698 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet cbba240087984f5bac2370d46a09a436 from peer c6f646ec3edd4e298f16122028bb0ef7 at {username='slave'} at 127.0.99.133:51435: session id = c6f646ec3edd4e298f16122028bb0ef7-cbba240087984f5bac2370d46a09a436
I20240503 01:59:11.946306 3698 tablet_copy_source_session.cc:205] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:11.948526 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbba240087984f5bac2370d46a09a436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.949563 5610 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=717 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:11.949894 5610 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Bootstrap complete.
I20240503 01:59:11.950229 5610 ts_tablet_manager.cc:1367] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Time spent bootstrapping tablet: real 0.081s user 0.060s sys 0.004s
I20240503 01:59:11.953006 5630 tablet_copy_client.cc:783] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:11.953204 5630 tablet_copy_client.cc:647] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:11.954969 5610 raft_consensus.cc:348] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:11.955222 5630 tablet_copy_client.cc:523] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:11.955260 5610 raft_consensus.cc:729] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a367744629a43edb0df92463a498258, State: Initialized, Role: LEARNER
I20240503 01:59:11.955696 5610 consensus_queue.cc:260] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:11.956815 5610 ts_tablet_manager.cc:1398] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 01:59:11.957155 5630 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap starting.
I20240503 01:59:11.958581 3531 catalog_manager.cc:5484] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index -1 to 52, NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.959829 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 7a367744629a43edb0df92463a498258-287bd1cfeb1e4e709d40fb3368aaca77 received from {username='slave'} at 127.0.99.134:46137
I20240503 01:59:11.959909 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 7a367744629a43edb0df92463a498258-287bd1cfeb1e4e709d40fb3368aaca77 on tablet 287bd1cfeb1e4e709d40fb3368aaca77 with peer 7a367744629a43edb0df92463a498258
I20240503 01:59:11.979470 3809 consensus_queue.cc:237] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:11.981792 3940 raft_consensus.cc:1261] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:11.982318 4734 consensus_queue.cc:1035] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:11.985265 5423 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:11.986547 3940 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
W20240503 01:59:12.002017 3745 consensus_peers.cc:485] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 -> Peer c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793): Couldn't send request to peer c6f646ec3edd4e298f16122028bb0ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02a421366d744afd99d7779644e9e41e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:12.023238 5054 raft_consensus.cc:1261] T ea7e73129def488bbf9ed086e424968e P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Refusing update from remote peer 7b5c836ac65b442e808f46fb2d267d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 01:59:12.024385 5644 consensus_queue.cc:1035] T ea7e73129def488bbf9ed086e424968e P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.041859 3531 catalog_manager.cc:5484] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index -1 to 59, NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } } }
I20240503 01:59:12.050148 5610 ts_tablet_manager.cc:902] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:12.050684 5610 tablet_copy_client.cc:308] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:12.063241 3698 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet f8d39433e68b4bd7b14b42e850f42ae6 from peer 7a367744629a43edb0df92463a498258 ({username='slave'} at 127.0.99.134:41769)
I20240503 01:59:12.063383 3698 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet f8d39433e68b4bd7b14b42e850f42ae6 from peer 7a367744629a43edb0df92463a498258 at {username='slave'} at 127.0.99.134:41769: session id = 7a367744629a43edb0df92463a498258-f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:12.064460 3698 tablet_copy_source_session.cc:205] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.065099 5630 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.065497 5630 tablet_bootstrap.cc:492] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap complete.
I20240503 01:59:12.065726 5630 ts_tablet_manager.cc:1367] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Time spent bootstrapping tablet: real 0.109s user 0.075s sys 0.007s
I20240503 01:59:12.066360 5630 raft_consensus.cc:348] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.066540 5610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d39433e68b4bd7b14b42e850f42ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.066555 5630 raft_consensus.cc:729] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Initialized, Role: LEARNER
I20240503 01:59:12.066679 5630 consensus_queue.cc:260] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.068881 5610 tablet_copy_client.cc:783] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.069067 5610 tablet_copy_client.cc:647] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.069737 5630 ts_tablet_manager.cc:1398] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:12.070458 3698 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session c6f646ec3edd4e298f16122028bb0ef7-cbba240087984f5bac2370d46a09a436 received from {username='slave'} at 127.0.99.133:51435
I20240503 01:59:12.070547 3698 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session c6f646ec3edd4e298f16122028bb0ef7-cbba240087984f5bac2370d46a09a436 on tablet cbba240087984f5bac2370d46a09a436 with peer c6f646ec3edd4e298f16122028bb0ef7
I20240503 01:59:12.081620 5662 ts_tablet_manager.cc:902] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Initiating tablet copy from peer f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:12.083468 5662 tablet_copy_client.cc:308] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:35911
I20240503 01:59:12.090552 5318 raft_consensus.cc:1203] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.55] Dedup: 1.55->[]
I20240503 01:59:12.100744 3829 tablet_copy_service.cc:140] P f929df19fdfc4df1986e990c60056733: Received BeginTabletCopySession request for tablet 6365fecf9fa04d6eaf6ce9dbe47add84 from peer 7b5c836ac65b442e808f46fb2d267d2c ({username='slave'} at 127.0.99.132:40271)
I20240503 01:59:12.100910 3829 tablet_copy_service.cc:161] P f929df19fdfc4df1986e990c60056733: Beginning new tablet copy session on tablet 6365fecf9fa04d6eaf6ce9dbe47add84 from peer 7b5c836ac65b442e808f46fb2d267d2c at {username='slave'} at 127.0.99.132:40271: session id = 7b5c836ac65b442e808f46fb2d267d2c-6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:12.101857 5610 tablet_copy_client.cc:523] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.103292 5610 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Bootstrap starting.
I20240503 01:59:12.136672 5452 raft_consensus.cc:1050] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 to VOTER
I20240503 01:59:12.137091 3829 tablet_copy_source_session.cc:205] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.137069 5452 consensus_queue.cc:237] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:12.139637 3677 consensus_queue.cc:237] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.140144 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6365fecf9fa04d6eaf6ce9dbe47add84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.141999 3807 raft_consensus.cc:1261] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:12.142553 5433 consensus_queue.cc:1035] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:12.143275 3807 raft_consensus.cc:1261] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240503 01:59:12.144415 4834 consensus_queue.cc:1035] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 01:59:12.144845 5452 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 40 to 59, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.145900 3806 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 40 to 59, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.147742 3531 catalog_manager.cc:5484] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 40 to 59, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.149794 5662 tablet_copy_client.cc:783] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.150024 5662 tablet_copy_client.cc:647] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.152330 4834 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index -1 to 73, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:12.153262 5594 ts_tablet_manager.cc:902] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Initiating tablet copy from peer f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:12.153339 5317 raft_consensus.cc:1261] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 01:59:12.153900 5594 tablet_copy_client.cc:308] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:35911
I20240503 01:59:12.154479 3807 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index -1 to 73, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:12.154695 5447 consensus_queue.cc:1035] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 01:59:12.155913 5317 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 40 to 59, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.158177 5662 tablet_copy_client.cc:523] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.162919 3534 catalog_manager.cc:5484] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index -1 to 73, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:12.164168 3829 tablet_copy_service.cc:140] P f929df19fdfc4df1986e990c60056733: Received BeginTabletCopySession request for tablet e6c983adafca4912aa70f99c81bb4b4f from peer 0fb6a1bb16214e3c9ec6acd1ab111358 ({username='slave'} at 127.0.99.135:41767)
I20240503 01:59:12.164297 3829 tablet_copy_service.cc:161] P f929df19fdfc4df1986e990c60056733: Beginning new tablet copy session on tablet e6c983adafca4912aa70f99c81bb4b4f from peer 0fb6a1bb16214e3c9ec6acd1ab111358 at {username='slave'} at 127.0.99.135:41767: session id = 0fb6a1bb16214e3c9ec6acd1ab111358-e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:12.165899 3829 tablet_copy_source_session.cc:205] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.167714 5594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c983adafca4912aa70f99c81bb4b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.168709 5662 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap starting.
I20240503 01:59:12.170038 5594 tablet_copy_client.cc:783] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.170176 5594 tablet_copy_client.cc:647] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 1 WAL segments...
W20240503 01:59:12.184926 3613 consensus_peers.cc:485] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b -> Peer 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): Couldn't send request to peer 0fb6a1bb16214e3c9ec6acd1ab111358. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d2ab9fbe93a44adbc34f306e8d61c27. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:12.205323 5594 tablet_copy_client.cc:523] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.206868 5594 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap starting.
I20240503 01:59:12.221421 5610 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=748 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.221765 5610 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Bootstrap complete.
I20240503 01:59:12.221985 5610 ts_tablet_manager.cc:1367] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Time spent bootstrapping tablet: real 0.119s user 0.055s sys 0.016s
I20240503 01:59:12.222587 5610 raft_consensus.cc:348] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:12.222728 5610 raft_consensus.cc:729] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a367744629a43edb0df92463a498258, State: Initialized, Role: LEARNER
I20240503 01:59:12.222836 5610 consensus_queue.cc:260] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:12.224895 5610 ts_tablet_manager.cc:1398] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:12.227942 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 7a367744629a43edb0df92463a498258-f8d39433e68b4bd7b14b42e850f42ae6 received from {username='slave'} at 127.0.99.134:41769
I20240503 01:59:12.228030 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 7a367744629a43edb0df92463a498258-f8d39433e68b4bd7b14b42e850f42ae6 on tablet f8d39433e68b4bd7b14b42e850f42ae6 with peer 7a367744629a43edb0df92463a498258
I20240503 01:59:12.229977 3939 consensus_queue.cc:237] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:12.231153 5318 raft_consensus.cc:1261] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 01:59:12.231887 5452 consensus_queue.cc:1035] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 01:59:12.234114 5452 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.236474 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c6b5cb588234440807eee9875c334c1 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.238353 3534 catalog_manager.cc:5484] T 1c6b5cb588234440807eee9875c334c1 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 59 to 61, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.241055 5318 raft_consensus.cc:2932] T 1c6b5cb588234440807eee9875c334c1 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.246428 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6b5cb588234440807eee9875c334c1 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 61) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:12.246981 5699 tablet_replica.cc:330] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:12.247393 5699 raft_consensus.cc:2229] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.247745 5699 raft_consensus.cc:2258] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.249001 5630 ts_tablet_manager.cc:902] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:12.249547 5630 tablet_copy_client.cc:308] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:12.251421 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet e504950788d7408a83d4b7299b78518c from peer c6f646ec3edd4e298f16122028bb0ef7 ({username='slave'} at 127.0.99.133:51435)
I20240503 01:59:12.251583 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet e504950788d7408a83d4b7299b78518c from peer c6f646ec3edd4e298f16122028bb0ef7 at {username='slave'} at 127.0.99.133:51435: session id = c6f646ec3edd4e298f16122028bb0ef7-e504950788d7408a83d4b7299b78518c
I20240503 01:59:12.252427 5699 ts_tablet_manager.cc:1875] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.253260 3697 tablet_copy_source_session.cc:205] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.255692 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e504950788d7408a83d4b7299b78518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.258116 5630 tablet_copy_client.cc:783] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.258314 5630 tablet_copy_client.cc:647] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.259500 5699 ts_tablet_manager.cc:1888] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.60
I20240503 01:59:12.259641 5699 log.cc:1196] T 1c6b5cb588234440807eee9875c334c1 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c6b5cb588234440807eee9875c334c1
I20240503 01:59:12.261513 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet 1c6b5cb588234440807eee9875c334c1 (table rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]) successfully deleted
I20240503 01:59:12.264977 5187 raft_consensus.cc:1203] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.58] Dedup: 1.45->[1.46-1.58]
I20240503 01:59:12.279894 5630 tablet_copy_client.cc:523] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.283697 5630 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap starting.
I20240503 01:59:12.307266 5610 ts_tablet_manager.cc:902] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:12.308480 5610 tablet_copy_client.cc:308] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:12.308564 5706 ts_tablet_manager.cc:902] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:12.309108 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet 5dba64ef1acb4293be37a986fc18a105 from peer 7a367744629a43edb0df92463a498258 ({username='slave'} at 127.0.99.134:41769)
I20240503 01:59:12.309243 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet 5dba64ef1acb4293be37a986fc18a105 from peer 7a367744629a43edb0df92463a498258 at {username='slave'} at 127.0.99.134:41769: session id = 7a367744629a43edb0df92463a498258-5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:12.309265 5706 tablet_copy_client.cc:308] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:12.309720 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet 56cd6bd86d2a4391947f7b7193b5e420 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 ({username='slave'} at 127.0.99.135:48497)
I20240503 01:59:12.309832 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet 56cd6bd86d2a4391947f7b7193b5e420 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 at {username='slave'} at 127.0.99.135:48497: session id = 0fb6a1bb16214e3c9ec6acd1ab111358-56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:12.310249 3697 tablet_copy_source_session.cc:205] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.310822 3960 tablet_copy_source_session.cc:205] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.312849 5610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba64ef1acb4293be37a986fc18a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.315301 5610 tablet_copy_client.cc:783] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.315490 5610 tablet_copy_client.cc:647] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.326313 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cd6bd86d2a4391947f7b7193b5e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.332342 5706 tablet_copy_client.cc:783] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.332551 5706 tablet_copy_client.cc:647] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.334664 5610 tablet_copy_client.cc:523] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.336130 5610 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Bootstrap starting.
I20240503 01:59:12.337317 5662 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=759 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.337669 5662 tablet_bootstrap.cc:492] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap complete.
I20240503 01:59:12.337896 5662 ts_tablet_manager.cc:1367] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Time spent bootstrapping tablet: real 0.169s user 0.090s sys 0.007s
I20240503 01:59:12.338546 5662 raft_consensus.cc:348] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:12.338680 5662 raft_consensus.cc:729] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Initialized, Role: LEARNER
I20240503 01:59:12.338809 5662 consensus_queue.cc:260] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:12.338929 5452 raft_consensus.cc:1050] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 7a367744629a43edb0df92463a498258 to VOTER
I20240503 01:59:12.341123 5662 ts_tablet_manager.cc:1398] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:12.341194 5452 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.342396 5187 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 01:59:12.342446 3806 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 01:59:12.342804 4836 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 01:59:12.342968 5447 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 01:59:12.343154 3829 tablet_copy_service.cc:342] P f929df19fdfc4df1986e990c60056733: Request end of tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-6365fecf9fa04d6eaf6ce9dbe47add84 received from {username='slave'} at 127.0.99.132:40271
I20240503 01:59:12.343251 3829 tablet_copy_service.cc:433] P f929df19fdfc4df1986e990c60056733: ending tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-6365fecf9fa04d6eaf6ce9dbe47add84 on tablet 6365fecf9fa04d6eaf6ce9dbe47add84 with peer 7b5c836ac65b442e808f46fb2d267d2c
I20240503 01:59:12.345060 5447 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 29 to 60, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.345779 5662 ts_tablet_manager.cc:902] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:12.346282 5662 tablet_copy_client.cc:308] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:12.348341 5187 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 29 to 60, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.350044 3531 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 29 to 60, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.352043 3806 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 29 to 60, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.360608 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet 6a7cf094bb2546bb800180d3a530f46a from peer 7b5c836ac65b442e808f46fb2d267d2c ({username='slave'} at 127.0.99.132:39973)
I20240503 01:59:12.360713 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet 6a7cf094bb2546bb800180d3a530f46a from peer 7b5c836ac65b442e808f46fb2d267d2c at {username='slave'} at 127.0.99.132:39973: session id = 7b5c836ac65b442e808f46fb2d267d2c-6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:12.361639 3697 tablet_copy_source_session.cc:205] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.363663 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7cf094bb2546bb800180d3a530f46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.365523 5706 tablet_copy_client.cc:523] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.365891 5662 tablet_copy_client.cc:783] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.366034 5662 tablet_copy_client.cc:647] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.367058 5706 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap starting.
I20240503 01:59:12.370013 5662 tablet_copy_client.cc:523] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.371203 5662 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap starting.
I20240503 01:59:12.373610 5594 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.374524 5594 tablet_bootstrap.cc:492] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap complete.
I20240503 01:59:12.374791 5594 ts_tablet_manager.cc:1367] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent bootstrapping tablet: real 0.168s user 0.068s sys 0.004s
I20240503 01:59:12.375440 5594 raft_consensus.cc:348] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.375691 5594 raft_consensus.cc:729] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Initialized, Role: LEARNER
I20240503 01:59:12.375860 5594 consensus_queue.cc:260] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.376338 5594 ts_tablet_manager.cc:1398] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.377631 3829 tablet_copy_service.cc:342] P f929df19fdfc4df1986e990c60056733: Request end of tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-e6c983adafca4912aa70f99c81bb4b4f received from {username='slave'} at 127.0.99.135:41767
I20240503 01:59:12.377710 3829 tablet_copy_service.cc:433] P f929df19fdfc4df1986e990c60056733: ending tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-e6c983adafca4912aa70f99c81bb4b4f on tablet e6c983adafca4912aa70f99c81bb4b4f with peer 0fb6a1bb16214e3c9ec6acd1ab111358
I20240503 01:59:12.393414 3939 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46664
I20240503 01:59:12.393630 3939 raft_consensus.cc:595] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:12.449689 5610 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=65 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:12.450248 5610 tablet_bootstrap.cc:492] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Bootstrap complete.
I20240503 01:59:12.450512 5610 ts_tablet_manager.cc:1367] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Time spent bootstrapping tablet: real 0.114s user 0.071s sys 0.003s
I20240503 01:59:12.451090 5610 raft_consensus.cc:348] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:12.451625 5610 raft_consensus.cc:729] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a367744629a43edb0df92463a498258, State: Initialized, Role: LEARNER
I20240503 01:59:12.451809 5610 consensus_queue.cc:260] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:12.452988 5610 ts_tablet_manager.cc:1398] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.454478 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 7a367744629a43edb0df92463a498258-5dba64ef1acb4293be37a986fc18a105 received from {username='slave'} at 127.0.99.134:41769
I20240503 01:59:12.454552 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 7a367744629a43edb0df92463a498258-5dba64ef1acb4293be37a986fc18a105 on tablet 5dba64ef1acb4293be37a986fc18a105 with peer 7a367744629a43edb0df92463a498258
I20240503 01:59:12.473553 5054 raft_consensus.cc:1203] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.53->[1.54-1.79] Dedup: 1.70->[1.71-1.79]
I20240503 01:59:12.473974 5723 ts_tablet_manager.cc:902] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:12.474649 5723 tablet_copy_client.cc:308] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:12.478929 5187 raft_consensus.cc:1203] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.41->[1.42-1.67] Dedup: 1.63->[1.64-1.67]
I20240503 01:59:12.488386 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 from peer 7b5c836ac65b442e808f46fb2d267d2c ({username='slave'} at 127.0.99.132:58941)
I20240503 01:59:12.488503 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 from peer 7b5c836ac65b442e808f46fb2d267d2c at {username='slave'} at 127.0.99.132:58941: session id = 7b5c836ac65b442e808f46fb2d267d2c-a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:12.489408 3960 tablet_copy_source_session.cc:205] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.490360 5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.492630 5723 tablet_copy_client.cc:783] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.492786 5723 tablet_copy_client.cc:647] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.494372 5662 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=741 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.494625 5662 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap complete.
I20240503 01:59:12.494786 5662 ts_tablet_manager.cc:1367] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Time spent bootstrapping tablet: real 0.124s user 0.059s sys 0.005s
I20240503 01:59:12.495245 5662 raft_consensus.cc:348] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:12.495352 5662 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Initialized, Role: LEARNER
I20240503 01:59:12.495452 5662 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
W20240503 01:59:12.497059 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
W20240503 01:59:12.501921 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.502039 5723 tablet_copy_client.cc:523] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.503451 5723 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap starting.
I20240503 01:59:12.509070 5384 raft_consensus.cc:1050] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 to VOTER
W20240503 01:59:12.510605 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.513543 5384 consensus_queue.cc:237] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:12.513913 5662 ts_tablet_manager.cc:1398] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240503 01:59:12.516834 3939 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
W20240503 01:59:12.517551 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.519081 4919 raft_consensus.cc:1203] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.83] Dedup: 1.74->[1.75-1.83]
W20240503 01:59:12.525977 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.527077 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-6a7cf094bb2546bb800180d3a530f46a received from {username='slave'} at 127.0.99.132:39973
I20240503 01:59:12.527170 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-6a7cf094bb2546bb800180d3a530f46a on tablet 6a7cf094bb2546bb800180d3a530f46a with peer 7b5c836ac65b442e808f46fb2d267d2c
I20240503 01:59:12.527627 5515 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 01:59:12.529672 5406 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 54 to 82, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.530742 5630 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=703 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.531090 5630 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap complete.
I20240503 01:59:12.531260 5630 ts_tablet_manager.cc:1367] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Time spent bootstrapping tablet: real 0.248s user 0.073s sys 0.003s
I20240503 01:59:12.531787 5630 raft_consensus.cc:348] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.531984 5630 raft_consensus.cc:729] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Initialized, Role: LEARNER
I20240503 01:59:12.532100 5630 consensus_queue.cc:260] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.532788 3939 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 54 to 82, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.534202 5594 ts_tablet_manager.cc:902] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
W20240503 01:59:12.534600 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.534708 5594 tablet_copy_client.cc:308] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:12.536387 3529 catalog_manager.cc:5484] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 54 to 82, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.536933 5053 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 01:59:12.537242 5630 ts_tablet_manager.cc:1398] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:12.539062 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session c6f646ec3edd4e298f16122028bb0ef7-e504950788d7408a83d4b7299b78518c received from {username='slave'} at 127.0.99.133:51435
I20240503 01:59:12.539144 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session c6f646ec3edd4e298f16122028bb0ef7-e504950788d7408a83d4b7299b78518c on tablet e504950788d7408a83d4b7299b78518c with peer c6f646ec3edd4e298f16122028bb0ef7
I20240503 01:59:12.540962 5406 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 01:59:12.542616 5053 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 54 to 82, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.546085 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet 4d2ab9fbe93a44adbc34f306e8d61c27 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 ({username='slave'} at 127.0.99.135:52345)
I20240503 01:59:12.546180 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet 4d2ab9fbe93a44adbc34f306e8d61c27 from peer 0fb6a1bb16214e3c9ec6acd1ab111358 at {username='slave'} at 127.0.99.135:52345: session id = 0fb6a1bb16214e3c9ec6acd1ab111358-4d2ab9fbe93a44adbc34f306e8d61c27
W20240503 01:59:12.547075 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.547127 3697 tablet_copy_source_session.cc:205] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.548188 5594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2ab9fbe93a44adbc34f306e8d61c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.550395 5594 tablet_copy_client.cc:783] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.550541 5594 tablet_copy_client.cc:647] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.561427 5706 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=67 ignored=0} inserts{seen=698 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:12.561817 5706 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap complete.
W20240503 01:59:12.561868 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.562053 5516 raft_consensus.cc:1050] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 7a367744629a43edb0df92463a498258 to VOTER
I20240503 01:59:12.562331 5516 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.563619 5706 ts_tablet_manager.cc:1367] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent bootstrapping tablet: real 0.197s user 0.063s sys 0.007s
I20240503 01:59:12.564613 5706 raft_consensus.cc:348] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.566012 3806 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 01:59:12.566867 5594 tablet_copy_client.cc:523] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.567016 5384 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:12.567670 5706 raft_consensus.cc:729] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Initialized, Role: LEARNER
I20240503 01:59:12.567795 5706 consensus_queue.cc:260] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.568115 5706 ts_tablet_manager.cc:1398] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:12.568691 5594 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap starting.
I20240503 01:59:12.570122 5187 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 01:59:12.571727 5384 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 01:59:12.572608 5630 ts_tablet_manager.cc:902] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Initiating tablet copy from peer f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:12.573064 5630 tablet_copy_client.cc:308] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:35911
I20240503 01:59:12.573844 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-56cd6bd86d2a4391947f7b7193b5e420 received from {username='slave'} at 127.0.99.135:48497
I20240503 01:59:12.573927 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-56cd6bd86d2a4391947f7b7193b5e420 on tablet 56cd6bd86d2a4391947f7b7193b5e420 with peer 0fb6a1bb16214e3c9ec6acd1ab111358
I20240503 01:59:12.576313 5516 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index 42 to 69, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.578979 3806 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 42 to 69, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.585395 5187 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 42 to 69, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.586752 3829 tablet_copy_service.cc:140] P f929df19fdfc4df1986e990c60056733: Received BeginTabletCopySession request for tablet 02a421366d744afd99d7779644e9e41e from peer c6f646ec3edd4e298f16122028bb0ef7 ({username='slave'} at 127.0.99.133:48235)
I20240503 01:59:12.586863 3829 tablet_copy_service.cc:161] P f929df19fdfc4df1986e990c60056733: Beginning new tablet copy session on tablet 02a421366d744afd99d7779644e9e41e from peer c6f646ec3edd4e298f16122028bb0ef7 at {username='slave'} at 127.0.99.133:48235: session id = c6f646ec3edd4e298f16122028bb0ef7-02a421366d744afd99d7779644e9e41e
W20240503 01:59:12.587762 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.588042 3829 tablet_copy_source_session.cc:205] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:12.588188 3529 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 42 to 69, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.589038 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a421366d744afd99d7779644e9e41e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.591183 5630 tablet_copy_client.cc:783] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:12.591322 5630 tablet_copy_client.cc:647] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:12.592000 5423 raft_consensus.cc:1050] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: attempting to promote NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c to VOTER
I20240503 01:59:12.592422 5423 consensus_queue.cc:237] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:12.594157 5630 tablet_copy_client.cc:523] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:12.595618 5630 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap starting.
I20240503 01:59:12.595578 3676 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.596709 4919 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 01:59:12.597141 5456 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 01:59:12.599326 3678 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 01:59:12.600140 5456 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
W20240503 01:59:12.604988 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.611673 5456 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 55 to 85, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:12.612562 5187 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 01:59:12.613091 5406 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 01:59:12.613400 3677 raft_consensus.cc:1203] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.83->[1.84-1.84] Dedup: 1.84->[]
I20240503 01:59:12.614100 5723 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=68 ignored=0} inserts{seen=699 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:12.614354 5723 tablet_bootstrap.cc:492] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Bootstrap complete.
I20240503 01:59:12.614498 5723 ts_tablet_manager.cc:1367] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Time spent bootstrapping tablet: real 0.111s user 0.068s sys 0.000s
I20240503 01:59:12.614949 5723 raft_consensus.cc:348] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:12.615511 5723 raft_consensus.cc:729] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Initialized, Role: LEARNER
I20240503 01:59:12.615666 5723 consensus_queue.cc:260] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: true } }
I20240503 01:59:12.615880 3678 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 55 to 85, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:12.616501 5723 ts_tablet_manager.cc:1398] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:12.619163 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-a67fef22e6eb4ecebb2ba7c4d7a5b0c5 received from {username='slave'} at 127.0.99.132:58941
I20240503 01:59:12.619247 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 7b5c836ac65b442e808f46fb2d267d2c-a67fef22e6eb4ecebb2ba7c4d7a5b0c5 on tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 with peer 7b5c836ac65b442e808f46fb2d267d2c
W20240503 01:59:12.623353 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.623988 3531 catalog_manager.cc:5484] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 55 to 85, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
W20240503 01:59:12.636276 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.639737 3678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbba240087984f5bac2370d46a09a436"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60762
I20240503 01:59:12.639922 3678 raft_consensus.cc:595] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:12.642830 5515 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 69 to 70, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.644888 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8d39433e68b4bd7b14b42e850f42ae6 with cas_config_opid_index 69: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.646826 5187 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 69 to 70, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.646819 3534 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 69 to 70, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.652654 4919 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 55 to 85, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
W20240503 01:59:12.654254 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
W20240503 01:59:12.660023 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.665365 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d39433e68b4bd7b14b42e850f42ae6 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:12.665616 5699 tablet_replica.cc:330] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:12.665750 5699 raft_consensus.cc:2229] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.665907 5699 raft_consensus.cc:2258] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.666957 5699 ts_tablet_manager.cc:1875] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.669210 5699 ts_tablet_manager.cc:1888] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.69
I20240503 01:59:12.669283 5699 log.cc:1196] T f8d39433e68b4bd7b14b42e850f42ae6 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8d39433e68b4bd7b14b42e850f42ae6
W20240503 01:59:12.671206 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.672528 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet f8d39433e68b4bd7b14b42e850f42ae6 (table rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]) successfully deleted
W20240503 01:59:12.673372 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
W20240503 01:59:12.675530 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.676329 5318 raft_consensus.cc:1203] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.76] Dedup: 1.68->[1.69-1.76]
W20240503 01:59:12.679659 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.683940 5630 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.684208 5630 tablet_bootstrap.cc:492] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Bootstrap complete.
I20240503 01:59:12.684363 5630 ts_tablet_manager.cc:1367] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Time spent bootstrapping tablet: real 0.089s user 0.065s sys 0.008s
I20240503 01:59:12.684777 5630 raft_consensus.cc:348] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.684897 5630 raft_consensus.cc:729] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Initialized, Role: LEARNER
I20240503 01:59:12.684993 5630 consensus_queue.cc:260] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: true } }
I20240503 01:59:12.685205 5630 ts_tablet_manager.cc:1398] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:12.689792 3829 tablet_copy_service.cc:342] P f929df19fdfc4df1986e990c60056733: Request end of tablet copy session c6f646ec3edd4e298f16122028bb0ef7-02a421366d744afd99d7779644e9e41e received from {username='slave'} at 127.0.99.133:48235
I20240503 01:59:12.689875 3829 tablet_copy_service.cc:433] P f929df19fdfc4df1986e990c60056733: ending tablet copy session c6f646ec3edd4e298f16122028bb0ef7-02a421366d744afd99d7779644e9e41e on tablet 02a421366d744afd99d7779644e9e41e with peer c6f646ec3edd4e298f16122028bb0ef7
W20240503 01:59:12.691179 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.698208 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.698422 5318 raft_consensus.cc:1203] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.74] Dedup: 1.63->[1.64-1.74]
W20240503 01:59:12.701721 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.711441 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.713004 5594 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=761 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:12.713413 5594 tablet_bootstrap.cc:492] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap complete.
I20240503 01:59:12.713606 5594 ts_tablet_manager.cc:1367] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent bootstrapping tablet: real 0.145s user 0.064s sys 0.013s
I20240503 01:59:12.714074 5594 raft_consensus.cc:348] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.714218 5594 raft_consensus.cc:729] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Initialized, Role: LEARNER
I20240503 01:59:12.714320 5594 consensus_queue.cc:260] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:12.715987 5594 ts_tablet_manager.cc:1398] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:12.717031 5053 raft_consensus.cc:1203] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.72] Dedup: 1.72->[]
I20240503 01:59:12.717533 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-4d2ab9fbe93a44adbc34f306e8d61c27 received from {username='slave'} at 127.0.99.135:52345
I20240503 01:59:12.717603 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-4d2ab9fbe93a44adbc34f306e8d61c27 on tablet 4d2ab9fbe93a44adbc34f306e8d61c27 with peer 0fb6a1bb16214e3c9ec6acd1ab111358
W20240503 01:59:12.722463 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.732021 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.732812 5516 raft_consensus.cc:1050] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 to VOTER
I20240503 01:59:12.733170 5516 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:12.735401 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 01:59:12.736507 3938 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 01:59:12.736557 4834 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 01:59:12.737154 4834 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
W20240503 01:59:12.738076 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.740401 5384 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 62 to 74, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
W20240503 01:59:12.746474 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.746379 5054 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 62 to 74, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.745674 3938 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 62 to 74, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.749190 3531 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 reported cstate change: config changed from index 62 to 74, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
W20240503 01:59:12.758035 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
W20240503 01:59:12.765348 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.767903 4836 raft_consensus.cc:1050] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 to VOTER
I20240503 01:59:12.768059 3677 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:12.768206 4836 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:12.770561 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 01:59:12.771447 4834 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 01:59:12.771555 3677 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:12.772004 5433 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:12.774005 5516 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 74 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.774495 5054 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 74 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.775490 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e504950788d7408a83d4b7299b78518c with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.777290 3529 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 reported cstate change: config changed from index 74 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:12.777724 5317 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:12.778313 5447 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:12.779917 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet e504950788d7408a83d4b7299b78518c with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 76) from {username='slave'} at 127.0.0.1:46500
W20240503 01:59:12.780375 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.780483 5816 tablet_replica.cc:330] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
W20240503 01:59:12.781347 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.781301 5447 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 56 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.783792 3529 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 56 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.784986 5318 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 56 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.786552 3938 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:12.788972 5452 raft_consensus.cc:979] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: : Instructing follower 7a367744629a43edb0df92463a498258 to start an election
I20240503 01:59:12.789052 5452 raft_consensus.cc:1067] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Signalling peer 7a367744629a43edb0df92463a498258 to start an election
I20240503 01:59:12.789698 5318 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 01:59:12.790462 5187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77"
dest_uuid: "7a367744629a43edb0df92463a498258"
from {username='slave'} at 127.0.99.131:53945
I20240503 01:59:12.790684 5187 raft_consensus.cc:482] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:12.790822 5187 raft_consensus.cc:3037] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:12.791076 3806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84"
dest_uuid: "f929df19fdfc4df1986e990c60056733"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36090
I20240503 01:59:12.791214 3806 raft_consensus.cc:595] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:12.792351 5187 raft_consensus.cc:504] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
W20240503 01:59:12.797118 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
W20240503 01:59:12.799224 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.799860 5447 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 01:59:12.800292 5187 leader_election.cc:288] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [CANDIDATE]: Term 2 election: Requested vote from peers f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911), c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
W20240503 01:59:12.800500 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Service unavailable: leader transfer in progress
I20240503 01:59:12.780791 5816 raft_consensus.cc:2229] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.801810 5816 raft_consensus.cc:2258] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.802351 3677 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 56 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.803390 4836 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.803655 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77" candidate_uuid: "7a367744629a43edb0df92463a498258" candidate_term: 2 candidate_status { last_received { term: 1 index: 60 } } ignore_live_leader: true dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:12.803761 3938 raft_consensus.cc:3032] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:12.803823 3938 raft_consensus.cc:729] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:12.803951 3938 consensus_queue.cc:260] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.804142 3938 raft_consensus.cc:3037] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:12.805040 3938 raft_consensus.cc:2454] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a367744629a43edb0df92463a498258 in term 2.
I20240503 01:59:12.805338 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56cd6bd86d2a4391947f7b7193b5e420 with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 01:59:12.805776 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.806536 5121 leader_election.cc:302] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a367744629a43edb0df92463a498258, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:12.806811 5636 raft_consensus.cc:2781] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:12.806986 5636 raft_consensus.cc:686] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 LEADER]: Becoming Leader. State: Replica: 7a367744629a43edb0df92463a498258, State: Running, Role: LEADER
I20240503 01:59:12.807183 5816 ts_tablet_manager.cc:1875] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.807179 3534 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 79 to 81, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.807274 5636 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.809376 5317 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.810189 3529 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 reported cstate change: term changed from 1 to 2, leader changed from c8595f9ae8f644d6901791810d7b970e (127.0.99.131) to 7a367744629a43edb0df92463a498258 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.811493 5816 ts_tablet_manager.cc:1888] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240503 01:59:12.811609 5816 log.cc:1196] T e504950788d7408a83d4b7299b78518c P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e504950788d7408a83d4b7299b78518c
I20240503 01:59:12.812388 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet e504950788d7408a83d4b7299b78518c (table rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]) successfully deleted
W20240503 01:59:12.812583 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.813954 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cd6bd86d2a4391947f7b7193b5e420 with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:12.816941 5823 tablet_replica.cc:330] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:12.817360 5823 raft_consensus.cc:2229] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.817535 5823 raft_consensus.cc:2258] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.821018 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287bd1cfeb1e4e709d40fb3368aaca77" candidate_uuid: "7a367744629a43edb0df92463a498258" candidate_term: 2 candidate_status { last_received { term: 1 index: 60 } } ignore_live_leader: true dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:12.821142 3807 raft_consensus.cc:3037] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:12.821453 5823 ts_tablet_manager.cc:1875] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.822206 3807 raft_consensus.cc:2454] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a367744629a43edb0df92463a498258 in term 2.
W20240503 01:59:12.821146 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.823812 5823 ts_tablet_manager.cc:1888] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.80
I20240503 01:59:12.823908 5823 log.cc:1196] T 56cd6bd86d2a4391947f7b7193b5e420 P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/56cd6bd86d2a4391947f7b7193b5e420
W20240503 01:59:12.824625 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46600: Illegal state: replica c8595f9ae8f644d6901791810d7b970e is not leader of this config: current role FOLLOWER
W20240503 01:59:12.825765 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.828814 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet 56cd6bd86d2a4391947f7b7193b5e420 (table rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]) successfully deleted
W20240503 01:59:12.833529 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
W20240503 01:59:12.839726 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
W20240503 01:59:12.845726 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.848754 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
W20240503 01:59:12.852932 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36120: Illegal state: replica f929df19fdfc4df1986e990c60056733 is not leader of this config: current role FOLLOWER
I20240503 01:59:12.857534 4920 raft_consensus.cc:1203] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.46->[1.47-1.72] Dedup: 1.56->[1.57-1.72]
W20240503 01:59:12.861455 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
W20240503 01:59:12.862890 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.874858 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.875838 5384 raft_consensus.cc:1050] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c to VOTER
I20240503 01:59:12.876215 5384 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:12.881668 4919 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 01:59:12.882550 3938 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
W20240503 01:59:12.882719 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.883639 5516 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 01:59:12.884078 5516 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
W20240503 01:59:12.887398 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.889801 4834 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 47 to 75, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:12.891012 4919 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 47 to 75, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:12.892848 3939 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 2 index: 62. (index mismatch)
I20240503 01:59:12.894060 5822 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 01:59:12.894553 3531 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: config changed from index 47 to 75, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:12.896495 3807 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 2 index: 62. (index mismatch)
I20240503 01:59:12.897006 5822 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 01:59:12.898283 5705 mvcc.cc:205] Tried to move back new op lower bound from 7023417560640102400 to 7023417560301633536. Current Snapshot: MvccSnapshot[applied={T|T < 7023417560640102400}]
I20240503 01:59:12.899724 3939 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 47 to 75, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
W20240503 01:59:12.903092 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
W20240503 01:59:12.903092 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.904755 5399 mvcc.cc:205] Tried to move back new op lower bound from 7023417560640102400 to 7023417560301633536. Current Snapshot: MvccSnapshot[applied={T|T < 7023417560640102400 or (T in {7023417560640102400})}]
W20240503 01:59:12.916188 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.922324 5318 raft_consensus.cc:1203] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.72->[1.73-1.77] Dedup: 1.76->[1.77-1.77]
I20240503 01:59:12.927279 5187 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 2.62, Last appended by leader: 60, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.928632 3808 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 62. Preceding OpId from leader: term: 2 index: 63. (index mismatch)
W20240503 01:59:12.928871 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.929376 5822 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 01:59:12.931037 5187 raft_consensus.cc:1203] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.57->[1.58-1.80] Dedup: 1.66->[1.67-1.80]
I20240503 01:59:12.931361 5516 raft_consensus.cc:1050] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 to VOTER
I20240503 01:59:12.931665 5516 consensus_queue.cc:237] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
W20240503 01:59:12.933480 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.933622 5636 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 LEADER]: Committing config change with OpId 2.63: config changed from index 60 to 63, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.934832 3808 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Committing config change with OpId 2.63: config changed from index 60 to 63, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.936133 5318 raft_consensus.cc:1261] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:12.936141 3522 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 287bd1cfeb1e4e709d40fb3368aaca77 with cas_config_opid_index 60: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.939064 3534 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 reported cstate change: config changed from index 60 to 63, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.940131 5406 consensus_queue.cc:1035] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:12.940994 3807 raft_consensus.cc:1261] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:12.941473 5406 consensus_queue.cc:1035] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:12.942257 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 287bd1cfeb1e4e709d40fb3368aaca77 with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 63) from {username='slave'} at 127.0.0.1:46500
I20240503 01:59:12.942577 5816 tablet_replica.cc:330] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
I20240503 01:59:12.942715 5816 raft_consensus.cc:2229] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.942840 5816 raft_consensus.cc:2258] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.943755 5816 ts_tablet_manager.cc:1875] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.945365 5406 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 73 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.946014 5816 ts_tablet_manager.cc:1888] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.62
I20240503 01:59:12.946086 5816 log.cc:1196] T 287bd1cfeb1e4e709d40fb3368aaca77 P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:12.947855 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet 287bd1cfeb1e4e709d40fb3368aaca77 (table rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]) successfully deleted
I20240503 01:59:12.948918 3534 catalog_manager.cc:5484] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 73 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:12.950080 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.950615 3807 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 73 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
W20240503 01:59:12.954419 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.957518 5317 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 73 to 79, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.961162 3677 consensus_queue.cc:237] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:12.961539 5384 raft_consensus.cc:1050] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 7a367744629a43edb0df92463a498258 to VOTER
I20240503 01:59:12.961885 5384 consensus_queue.cc:237] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.963145 3807 raft_consensus.cc:1261] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 01:59:12.963295 5187 raft_consensus.cc:1261] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 01:59:12.963488 5516 consensus_queue.cc:1035] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 01:59:12.963668 5515 consensus_queue.cc:1035] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 01:59:12.965382 5316 raft_consensus.cc:1261] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 01:59:12.965790 5516 consensus_queue.cc:1035] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 01:59:12.966156 5516 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 58 to 81, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.967275 3807 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 58 to 81, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.968087 5187 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 58 to 81, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.968910 3531 catalog_manager.cc:5484] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 58 to 81, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 01:59:12.970039 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.972288 5054 raft_consensus.cc:1203] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.77] Dedup: 1.71->[1.72-1.77]
I20240503 01:59:12.973304 4834 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 79 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
W20240503 01:59:12.976387 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:12.976986 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4d2ab9fbe93a44adbc34f306e8d61c27 with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.977643 3677 consensus_queue.cc:237] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:12.978030 5316 raft_consensus.cc:2932] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 79 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:12.978804 3531 catalog_manager.cc:5484] T 4d2ab9fbe93a44adbc34f306e8d61c27 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 79 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.980361 5187 raft_consensus.cc:1261] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 01:59:12.980748 5384 consensus_queue.cc:1035] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 01:59:12.982414 5516 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.985095 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ab9fbe93a44adbc34f306e8d61c27 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:12.985298 5699 tablet_replica.cc:330] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:12.985458 5699 raft_consensus.cc:2229] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.985575 5699 pending_rounds.cc:62] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Trying to abort 1 pending ops.
I20240503 01:59:12.985646 5699 pending_rounds.cc:68] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Aborting op as it isn't in flight: id { term: 1 index: 81 } timestamp: 7023417560879550464 op_type: WRITE_OP write_request { tablet_id: "4d2ab9fbe93a44adbc34f306e8d61c27" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000q\016i\030\020/\330m\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a48065fad34e6744" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417560844763136 authz_token { token_data: "\010\232\214\321\261\006\"3\n\005slave\022*\n 886d332d90d54101ad91fcaa63185b7a\020\001\030\001 \001(\001" signature: "<redacted>""G\3618\340\363\034\360\226\246\351\247\260\332\032\335\231A\0107\224H\265O\271\277\211\314U\007\223\200\010\024@\031\364~\270Iv\005\374\356wMR\311\2022d2\256\037\201\037\221I\360u;h\273\350\236" signing_key_seq_num: 0 } } request_id { client_id: "7278ef31ed6f43aba3a7402b42318066" seq_no: 198 first_incomplete_seq_no: 198 attempt_no: 0 }
I20240503 01:59:12.985955 5699 raft_consensus.cc:2258] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.986457 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dba64ef1acb4293be37a986fc18a105 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:12.986721 5699 ts_tablet_manager.cc:1875] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.988087 3529 catalog_manager.cc:5484] T 5dba64ef1acb4293be37a986fc18a105 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 81 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.988898 5699 ts_tablet_manager.cc:1888] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240503 01:59:12.988970 5699 log.cc:1196] T 4d2ab9fbe93a44adbc34f306e8d61c27 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d2ab9fbe93a44adbc34f306e8d61c27
I20240503 01:59:12.989488 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet 4d2ab9fbe93a44adbc34f306e8d61c27 (table rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]) successfully deleted
W20240503 01:59:12.991134 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:12.991714 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dba64ef1acb4293be37a986fc18a105 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:12.992624 5187 raft_consensus.cc:2932] T 5dba64ef1acb4293be37a986fc18a105 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:12.993355 5699 tablet_replica.cc:330] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:12.993474 5699 raft_consensus.cc:2229] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:12.993881 5699 raft_consensus.cc:2258] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:12.997510 5699 ts_tablet_manager.cc:1875] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:12.998279 5515 raft_consensus.cc:979] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: : Instructing follower c6f646ec3edd4e298f16122028bb0ef7 to start an election
I20240503 01:59:12.998391 5384 raft_consensus.cc:1067] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Signalling peer c6f646ec3edd4e298f16122028bb0ef7 to start an election
I20240503 01:59:13.003422 5054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbba240087984f5bac2370d46a09a436"
dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7"
from {username='slave'} at 127.0.99.129:50303
I20240503 01:59:13.003680 5054 raft_consensus.cc:482] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 01:59:13.003685 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Service unavailable: leader transfer in progress
I20240503 01:59:13.003751 5054 raft_consensus.cc:3037] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.004868 5054 raft_consensus.cc:504] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.005336 5054 leader_election.cc:288] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [CANDIDATE]: Term 2 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705), 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:13.007843 3677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbba240087984f5bac2370d46a09a436" candidate_uuid: "c6f646ec3edd4e298f16122028bb0ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 85 } } ignore_live_leader: true dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:13.007970 3677 raft_consensus.cc:3032] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:13.008014 3677 raft_consensus.cc:729] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:13.008165 3677 consensus_queue.cc:260] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.008322 3677 raft_consensus.cc:3037] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.009362 3677 raft_consensus.cc:2454] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f646ec3edd4e298f16122028bb0ef7 in term 2.
W20240503 01:59:13.010473 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.010818 4987 leader_election.cc:302] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c6f646ec3edd4e298f16122028bb0ef7; no voters:
I20240503 01:59:13.011058 5651 raft_consensus.cc:2781] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:13.011219 5651 raft_consensus.cc:686] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 2 LEADER]: Becoming Leader. State: Replica: c6f646ec3edd4e298f16122028bb0ef7, State: Running, Role: LEADER
I20240503 01:59:13.011526 5651 consensus_queue.cc:237] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.013034 5699 ts_tablet_manager.cc:1888] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240503 01:59:13.013114 5699 log.cc:1196] T 5dba64ef1acb4293be37a986fc18a105 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:13.014195 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbba240087984f5bac2370d46a09a436" candidate_uuid: "c6f646ec3edd4e298f16122028bb0ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 85 } } ignore_live_leader: true dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:13.014308 3939 raft_consensus.cc:3037] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.014559 3534 catalog_manager.cc:5484] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 reported cstate change: term changed from 1 to 2, leader changed from 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) to c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c6f646ec3edd4e298f16122028bb0ef7" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.014654 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet 5dba64ef1acb4293be37a986fc18a105 (table rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]) successfully deleted
I20240503 01:59:13.015255 3939 raft_consensus.cc:2454] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f646ec3edd4e298f16122028bb0ef7 in term 2.
W20240503 01:59:13.030068 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60612: Illegal state: replica 2820cbd55ee1470880e35a8ab671563b is not leader of this config: current role FOLLOWER
W20240503 01:59:13.032214 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.036218 4919 raft_consensus.cc:1203] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.51->[1.52-1.75] Dedup: 1.69->[1.70-1.75]
W20240503 01:59:13.053244 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.069808 5447 raft_consensus.cc:1050] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 7b5c836ac65b442e808f46fb2d267d2c to VOTER
I20240503 01:59:13.070118 5447 consensus_queue.cc:237] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
W20240503 01:59:13.071391 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46718: Illegal state: replica c8595f9ae8f644d6901791810d7b970e is not leader of this config: current role FOLLOWER
I20240503 01:59:13.071825 3678 raft_consensus.cc:1261] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 01:59:13.073225 4920 raft_consensus.cc:1261] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 01:59:13.073563 5423 raft_consensus.cc:1050] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733: attempting to promote NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 to VOTER
I20240503 01:59:13.073642 4836 consensus_queue.cc:1035] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 01:59:13.073906 5423 consensus_queue.cc:237] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:13.073937 4836 consensus_queue.cc:1035] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 01:59:13.074548 3678 raft_consensus.cc:1203] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.78] Dedup: 1.77->[1.78-1.78]
I20240503 01:59:13.075106 3938 raft_consensus.cc:1261] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 01:59:13.075417 5316 raft_consensus.cc:1261] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 01:59:13.075479 5423 consensus_queue.cc:1035] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
W20240503 01:59:13.076339 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.077021 5433 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index 52 to 78, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.077570 4920 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 52 to 78, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.077834 5456 consensus_queue.cc:1035] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 01:59:13.079124 5423 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 40 to 75, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.080209 3534 catalog_manager.cc:5484] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: config changed from index 52 to 78, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.080451 3938 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 40 to 75, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.081923 3678 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 52 to 78, 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.084741 5316 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 40 to 75, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.084981 3939 consensus_queue.cc:237] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.087042 3529 catalog_manager.cc:5484] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 40 to 75, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.087877 4919 raft_consensus.cc:1261] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:13.088213 5447 consensus_queue.cc:1035] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 01:59:13.091499 5433 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 78 to 79, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.092082 4919 raft_consensus.cc:2932] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 78 to 79, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.094583 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 with cas_config_opid_index 78: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.095269 3531 catalog_manager.cc:5484] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: config changed from index 78 to 79, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
W20240503 01:59:13.098907 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.099925 3807 consensus_queue.cc:237] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:13.101902 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 79) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:13.102089 5823 tablet_replica.cc:330] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:13.102200 5823 raft_consensus.cc:2229] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.102334 5823 raft_consensus.cc:2258] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.103085 5823 ts_tablet_manager.cc:1875] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.103406 5317 raft_consensus.cc:1261] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 01:59:13.104456 5423 consensus_queue.cc:1035] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 01:59:13.105440 5823 ts_tablet_manager.cc:1888] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.78
I20240503 01:59:13.105507 5823 log.cc:1196] T a67fef22e6eb4ecebb2ba7c4d7a5b0c5 P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:13.106117 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5 (table rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]) successfully deleted
I20240503 01:59:13.107025 5456 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 75 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.109748 5317 raft_consensus.cc:2932] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 75 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.110729 3520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6c983adafca4912aa70f99c81bb4b4f with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.112049 3531 catalog_manager.cc:5484] T e6c983adafca4912aa70f99c81bb4b4f P 0fb6a1bb16214e3c9ec6acd1ab111358 reported cstate change: config changed from index 75 to 76, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:13.114827 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c983adafca4912aa70f99c81bb4b4f with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 76) from {username='slave'} at 127.0.0.1:46500
I20240503 01:59:13.115027 5816 tablet_replica.cc:330] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
I20240503 01:59:13.115145 5816 raft_consensus.cc:2229] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.115435 5816 raft_consensus.cc:2258] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.116277 5816 ts_tablet_manager.cc:1875] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.118492 5816 ts_tablet_manager.cc:1888] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.75
I20240503 01:59:13.118577 5816 log.cc:1196] T e6c983adafca4912aa70f99c81bb4b4f P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:13.119038 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet e6c983adafca4912aa70f99c81bb4b4f (table rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]) successfully deleted
W20240503 01:59:13.123982 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.143709 3678 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer c6f646ec3edd4e298f16122028bb0ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 2 index: 87. (index mismatch)
I20240503 01:59:13.144400 5859 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240503 01:59:13.146034 3938 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Refusing update from remote peer c6f646ec3edd4e298f16122028bb0ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 2 index: 87. (index mismatch)
I20240503 01:59:13.146541 5651 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
W20240503 01:59:13.151894 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.171875 5054 consensus_queue.cc:237] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 2.88, Last appended by leader: 85, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.173565 3938 raft_consensus.cc:1261] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Refusing update from remote peer c6f646ec3edd4e298f16122028bb0ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 88. Preceding OpId from leader: term: 2 index: 90. (index mismatch)
I20240503 01:59:13.174265 5651 consensus_queue.cc:1035] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 01:59:13.178726 5651 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P c6f646ec3edd4e298f16122028bb0ef7 [term 2 LEADER]: Committing config change with OpId 2.89: config changed from index 82 to 89, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.179371 3938 raft_consensus.cc:2932] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Committing config change with OpId 2.89: config changed from index 82 to 89, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.181401 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cbba240087984f5bac2370d46a09a436 with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.181866 3529 catalog_manager.cc:5484] T cbba240087984f5bac2370d46a09a436 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 82 to 89, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "c6f646ec3edd4e298f16122028bb0ef7" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.182588 3678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60824
I20240503 01:59:13.182703 3678 raft_consensus.cc:595] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:13.183431 5515 raft_consensus.cc:979] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: : Instructing follower c8595f9ae8f644d6901791810d7b970e to start an election
I20240503 01:59:13.183854 5515 raft_consensus.cc:1067] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Signalling peer c8595f9ae8f644d6901791810d7b970e to start an election
I20240503 01:59:13.184408 3938 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
from {username='slave'} at 127.0.99.129:39121
I20240503 01:59:13.184545 3938 raft_consensus.cc:482] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:13.184603 3938 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.185694 3938 raft_consensus.cc:504] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.185961 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet cbba240087984f5bac2370d46a09a436 with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 89) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:13.186142 5823 tablet_replica.cc:330] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:13.186283 5823 raft_consensus.cc:2229] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.186290 3938 leader_election.cc:288] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 2 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461), 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901)
I20240503 01:59:13.186384 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 2 candidate_status { last_received { term: 1 index: 92 } } ignore_live_leader: true dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:13.186494 3678 raft_consensus.cc:3032] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:13.186545 3678 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:13.186661 3678 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.186791 3678 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.187508 3678 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 2.
I20240503 01:59:13.188043 3874 leader_election.cc:302] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2820cbd55ee1470880e35a8ab671563b, c8595f9ae8f644d6901791810d7b970e; no voters:
W20240503 01:59:13.188061 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Service unavailable: leader transfer in progress
I20240503 01:59:13.188205 5433 raft_consensus.cc:2781] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:13.188282 5433 raft_consensus.cc:686] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Becoming Leader. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:13.188416 5433 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.186420 5823 raft_consensus.cc:2258] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.189473 5823 ts_tablet_manager.cc:1875] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.191327 5823 ts_tablet_manager.cc:1888] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.88
I20240503 01:59:13.191390 5823 log.cc:1196] T cbba240087984f5bac2370d46a09a436 P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cbba240087984f5bac2370d46a09a436
W20240503 01:59:13.191488 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60608: Illegal state: replica 2820cbd55ee1470880e35a8ab671563b is not leader of this config: current role FOLLOWER
I20240503 01:59:13.192402 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet cbba240087984f5bac2370d46a09a436 (table rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]) successfully deleted
I20240503 01:59:13.192881 3529 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e reported cstate change: term changed from 1 to 2, leader changed from 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) to c8595f9ae8f644d6901791810d7b970e (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:13.195897 4919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "c8595f9ae8f644d6901791810d7b970e" candidate_term: 2 candidate_status { last_received { term: 1 index: 92 } } ignore_live_leader: true dest_uuid: "7b5c836ac65b442e808f46fb2d267d2c"
I20240503 01:59:13.196043 4919 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.197099 4919 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8595f9ae8f644d6901791810d7b970e in term 2.
I20240503 01:59:13.199083 3678 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 2 index: 94. (index mismatch)
I20240503 01:59:13.199441 5447 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:13.200624 4919 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 2 index: 94. (index mismatch)
I20240503 01:59:13.201525 5447 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 01:59:13.205708 5533 raft_consensus.cc:979] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: : Instructing follower 7b5c836ac65b442e808f46fb2d267d2c to start an election
I20240503 01:59:13.205811 5533 raft_consensus.cc:1067] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Signalling peer 7b5c836ac65b442e808f46fb2d267d2c to start an election
I20240503 01:59:13.206364 4918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84"
dest_uuid: "7b5c836ac65b442e808f46fb2d267d2c"
from {username='slave'} at 127.0.99.130:34223
I20240503 01:59:13.206524 4918 raft_consensus.cc:482] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:13.206593 4918 raft_consensus.cc:3037] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.207638 4918 raft_consensus.cc:504] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.208169 4918 leader_election.cc:288] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461), f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911)
I20240503 01:59:13.208653 3806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 90 } } ignore_live_leader: true dest_uuid: "f929df19fdfc4df1986e990c60056733"
I20240503 01:59:13.208774 3806 raft_consensus.cc:3032] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:13.208812 3806 raft_consensus.cc:729] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f929df19fdfc4df1986e990c60056733, State: Running, Role: LEADER
I20240503 01:59:13.208952 3806 consensus_queue.cc:260] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.209132 3806 raft_consensus.cc:3037] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.210050 3806 raft_consensus.cc:2454] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 2.
I20240503 01:59:13.210419 4852 leader_election.cc:302] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5c836ac65b442e808f46fb2d267d2c, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:13.210672 5652 raft_consensus.cc:2781] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:13.210845 5652 raft_consensus.cc:686] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 2 LEADER]: Becoming Leader. State: Replica: 7b5c836ac65b442e808f46fb2d267d2c, State: Running, Role: LEADER
I20240503 01:59:13.210978 5652 consensus_queue.cc:237] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.212692 5648 mvcc.cc:205] Tried to move back new op lower bound from 7023417561899438080 to 7023417561861144576. Current Snapshot: MvccSnapshot[applied={T|T < 7023417561899438080 or (T in {7023417561899438080})}]
I20240503 01:59:13.212772 3529 catalog_manager.cc:5484] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: term changed from 1 to 2, leader changed from f929df19fdfc4df1986e990c60056733 (127.0.99.130) to 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132). New cstate: current_term: 2 leader_uuid: "7b5c836ac65b442e808f46fb2d267d2c" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.213246 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6365fecf9fa04d6eaf6ce9dbe47add84" candidate_uuid: "7b5c836ac65b442e808f46fb2d267d2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 90 } } ignore_live_leader: true dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:13.213346 3678 raft_consensus.cc:3037] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:13.214309 3678 raft_consensus.cc:2454] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5c836ac65b442e808f46fb2d267d2c in term 2.
I20240503 01:59:13.215437 3938 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 2.94, Last appended by leader: 92, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.216677 4918 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 2 index: 94. Preceding OpId from leader: term: 2 index: 95. (index mismatch)
I20240503 01:59:13.217137 5433 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
W20240503 01:59:13.217379 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35998: Illegal state: replica f929df19fdfc4df1986e990c60056733 is not leader of this config: current role FOLLOWER
I20240503 01:59:13.219095 5433 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Committing config change with OpId 2.95: config changed from index 75 to 95, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.219378 4919 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Committing config change with OpId 2.95: config changed from index 75 to 95, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.220631 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a7cf094bb2546bb800180d3a530f46a with cas_config_opid_index 75: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.222373 3529 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 75 to 95, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.225031 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7cf094bb2546bb800180d3a530f46a with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:13.225237 5823 tablet_replica.cc:330] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:13.225356 5823 raft_consensus.cc:2229] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.225463 5823 raft_consensus.cc:2258] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.226135 5823 ts_tablet_manager.cc:1875] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.228298 5823 ts_tablet_manager.cc:1888] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.94
I20240503 01:59:13.228363 5823 log.cc:1196] T 6a7cf094bb2546bb800180d3a530f46a P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:13.228818 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet 6a7cf094bb2546bb800180d3a530f46a (table rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]) successfully deleted
W20240503 01:59:13.256982 3637 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60830: Illegal state: replica 2820cbd55ee1470880e35a8ab671563b is not leader of this config: current role FOLLOWER
I20240503 01:59:13.313150 3808 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Refusing update from remote peer 7b5c836ac65b442e808f46fb2d267d2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 2 index: 92. (index mismatch)
I20240503 01:59:13.313292 3678 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer 7b5c836ac65b442e808f46fb2d267d2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 2 index: 92. (index mismatch)
I20240503 01:59:13.314033 5891 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20240503 01:59:13.314435 5891 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20240503 01:59:13.330308 5423 raft_consensus.cc:1050] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733: attempting to promote NON_VOTER c6f646ec3edd4e298f16122028bb0ef7 to VOTER
I20240503 01:59:13.330752 5423 consensus_queue.cc:237] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.332192 3938 raft_consensus.cc:1261] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:13.332703 4734 consensus_queue.cc:1035] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 01:59:13.334259 5054 raft_consensus.cc:1261] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 LEARNER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 01:59:13.334254 4918 consensus_queue.cc:237] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 2.93, Last appended by leader: 90, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } }
I20240503 01:59:13.335196 5533 consensus_queue.cc:1035] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 01:59:13.335954 5533 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 59 to 79, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.336068 3938 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 59 to 79, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.336223 3678 raft_consensus.cc:1261] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer 7b5c836ac65b442e808f46fb2d267d2c: Log matching property violated. Preceding OpId in replica: term: 2 index: 93. Preceding OpId from leader: term: 2 index: 94. (index mismatch)
I20240503 01:59:13.337823 5906 consensus_queue.cc:1035] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240503 01:59:13.340238 5652 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c [term 2 LEADER]: Committing config change with OpId 2.94: config changed from index 85 to 94, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.342258 3529 catalog_manager.cc:5484] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index 59 to 79, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:13.343748 3534 catalog_manager.cc:5484] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: config changed from index 85 to 94, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "7b5c836ac65b442e808f46fb2d267d2c" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.345173 3520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6365fecf9fa04d6eaf6ce9dbe47add84 with cas_config_opid_index 85: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.347092 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 6365fecf9fa04d6eaf6ce9dbe47add84 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 94) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:13.347294 5699 tablet_replica.cc:330] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:13.347420 5699 raft_consensus.cc:2229] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.347532 5699 raft_consensus.cc:2258] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.348425 5699 ts_tablet_manager.cc:1875] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.350682 5699 ts_tablet_manager.cc:1888] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.93
I20240503 01:59:13.350749 5699 log.cc:1196] T 6365fecf9fa04d6eaf6ce9dbe47add84 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:13.351218 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet 6365fecf9fa04d6eaf6ce9dbe47add84 (table rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]) successfully deleted
I20240503 01:59:13.353928 5054 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 59 to 79, c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.370117 3808 consensus_queue.cc:237] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } }
I20240503 01:59:13.371439 5054 raft_consensus.cc:1261] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 01:59:13.371966 5533 consensus_queue.cc:1035] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 01:59:13.374293 5423 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.370510 3678 raft_consensus.cc:2932] T 6365fecf9fa04d6eaf6ce9dbe47add84 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Committing config change with OpId 2.94: config changed from index 85 to 94, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } }
I20240503 01:59:13.376551 5054 raft_consensus.cc:2932] T 02a421366d744afd99d7779644e9e41e P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } }
I20240503 01:59:13.377257 3531 catalog_manager.cc:5484] T 02a421366d744afd99d7779644e9e41e P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index 79 to 81, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.377951 3520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02a421366d744afd99d7779644e9e41e with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:13.383059 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a421366d744afd99d7779644e9e41e with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:46500
I20240503 01:59:13.383271 5816 tablet_replica.cc:330] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
I20240503 01:59:13.383410 5816 raft_consensus.cc:2229] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:13.383571 5816 raft_consensus.cc:2258] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:13.385080 5816 ts_tablet_manager.cc:1875] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:13.387190 5816 ts_tablet_manager.cc:1888] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.80
I20240503 01:59:13.387279 5816 log.cc:1196] T 02a421366d744afd99d7779644e9e41e P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/02a421366d744afd99d7779644e9e41e
I20240503 01:59:13.388507 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet 02a421366d744afd99d7779644e9e41e (table rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]) successfully deleted
I20240503 01:59:13.724826 3657 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:13.730650 3920 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:13.746598 5298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:13.766816 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.767016 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:13.785012 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.800161 5167 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:13.872399 3678 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:13.874302 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 01:59:13.874706 5515 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20240503 01:59:13.877355 5515 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 76 to 116, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:13.879621 5054 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 76 to 116, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } } }
I20240503 01:59:13.880278 3531 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 76 to 116, NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:13.887092 3613 consensus_peers.cc:485] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b -> Peer 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): Couldn't send request to peer 0fb6a1bb16214e3c9ec6acd1ab111358. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e504950788d7408a83d4b7299b78518c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:13.906235 3938 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 2.121, Last appended by leader: 92, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:13.908119 4918 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 2 index: 121. Preceding OpId from leader: term: 2 index: 122. (index mismatch)
I20240503 01:59:13.908435 5452 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
W20240503 01:59:13.908951 3875 consensus_peers.cc:485] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e -> Peer 7a367744629a43edb0df92463a498258 (127.0.99.134:42343): Couldn't send request to peer 7a367744629a43edb0df92463a498258. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a7cf094bb2546bb800180d3a530f46a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:13.909996 5433 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Committing config change with OpId 2.122: config changed from index 95 to 122, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:13.912124 4918 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Committing config change with OpId 2.122: config changed from index 95 to 122, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } } }
I20240503 01:59:13.913318 3529 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 95 to 122, NON_VOTER 7a367744629a43edb0df92463a498258 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:14.279701 6018 ts_tablet_manager.cc:902] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:14.280606 6018 tablet_copy_client.cc:308] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:14.281330 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet e504950788d7408a83d4b7299b78518c from peer 0fb6a1bb16214e3c9ec6acd1ab111358 ({username='slave'} at 127.0.99.135:52345)
I20240503 01:59:14.281427 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet e504950788d7408a83d4b7299b78518c from peer 0fb6a1bb16214e3c9ec6acd1ab111358 at {username='slave'} at 127.0.99.135:52345: session id = 0fb6a1bb16214e3c9ec6acd1ab111358-e504950788d7408a83d4b7299b78518c
I20240503 01:59:14.282327 3697 tablet_copy_source_session.cc:205] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:14.283597 6018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e504950788d7408a83d4b7299b78518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:14.285784 6018 tablet_copy_client.cc:783] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:14.285934 6018 tablet_copy_client.cc:647] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:14.294018 6018 tablet_copy_client.cc:523] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:14.295886 6018 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap starting.
I20240503 01:59:14.451884 6018 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap replayed 1/1 log segments. Stats: ops{read=124 overwritten=0 applied=124 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:14.452231 6018 tablet_bootstrap.cc:492] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Bootstrap complete.
I20240503 01:59:14.452459 6018 ts_tablet_manager.cc:1367] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent bootstrapping tablet: real 0.157s user 0.109s sys 0.004s
I20240503 01:59:14.452912 6018 raft_consensus.cc:348] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:14.453055 6018 raft_consensus.cc:729] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Initialized, Role: LEARNER
I20240503 01:59:14.453156 6018 consensus_queue.cc:260] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 124, Last appended: 1.124, Last appended by leader: 124, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: true } }
I20240503 01:59:14.454046 6018 ts_tablet_manager.cc:1398] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:14.459064 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-e504950788d7408a83d4b7299b78518c received from {username='slave'} at 127.0.99.135:52345
I20240503 01:59:14.459152 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 0fb6a1bb16214e3c9ec6acd1ab111358-e504950788d7408a83d4b7299b78518c on tablet e504950788d7408a83d4b7299b78518c with peer 0fb6a1bb16214e3c9ec6acd1ab111358
I20240503 01:59:14.505393 6036 ts_tablet_manager.cc:902] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:14.521711 6036 tablet_copy_client.cc:308] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:14.522187 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet 6a7cf094bb2546bb800180d3a530f46a from peer 7a367744629a43edb0df92463a498258 ({username='slave'} at 127.0.99.134:46137)
I20240503 01:59:14.522293 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet 6a7cf094bb2546bb800180d3a530f46a from peer 7a367744629a43edb0df92463a498258 at {username='slave'} at 127.0.99.134:46137: session id = 7a367744629a43edb0df92463a498258-6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:14.523240 3960 tablet_copy_source_session.cc:205] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:14.524637 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7cf094bb2546bb800180d3a530f46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:14.526930 6036 tablet_copy_client.cc:783] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:14.527072 6036 tablet_copy_client.cc:647] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:14.536708 6036 tablet_copy_client.cc:523] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:14.538268 6036 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Bootstrap starting.
I20240503 01:59:14.745348 6036 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Bootstrap replayed 1/1 log segments. Stats: ops{read=147 overwritten=0 applied=147 ignored=0} inserts{seen=828 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:14.745703 6036 tablet_bootstrap.cc:492] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Bootstrap complete.
I20240503 01:59:14.745913 6036 ts_tablet_manager.cc:1367] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Time spent bootstrapping tablet: real 0.208s user 0.115s sys 0.013s
I20240503 01:59:14.746387 6036 raft_consensus.cc:348] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:14.746562 6036 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7a367744629a43edb0df92463a498258, State: Initialized, Role: LEARNER
I20240503 01:59:14.746680 6036 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 147, Last appended: 2.147, Last appended by leader: 147, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: true } }
I20240503 01:59:14.747053 6036 ts_tablet_manager.cc:1398] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:14.747529 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 7a367744629a43edb0df92463a498258-6a7cf094bb2546bb800180d3a530f46a received from {username='slave'} at 127.0.99.134:46137
I20240503 01:59:14.747637 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 7a367744629a43edb0df92463a498258-6a7cf094bb2546bb800180d3a530f46a on tablet 6a7cf094bb2546bb800180d3a530f46a with peer 7a367744629a43edb0df92463a498258
I20240503 01:59:14.750838 5315 raft_consensus.cc:1203] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.115->[1.116-1.137] Dedup: 1.124->[1.125-1.137]
I20240503 01:59:14.804378 4834 raft_consensus.cc:1050] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 to VOTER
I20240503 01:59:14.804757 4834 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:14.805845 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:14.806526 5516 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 01:59:14.806649 5316 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 01:59:14.807523 5515 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 01:59:14.808089 5515 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index 116 to 140, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:14.810036 5054 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 116 to 140, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:14.812717 3529 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 reported cstate change: config changed from index 116 to 140, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:14.812830 5316 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 116 to 140, 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:15.057351 3677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e504950788d7408a83d4b7299b78518c"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60926
I20240503 01:59:15.057514 3677 raft_consensus.cc:595] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
W20240503 01:59:15.058661 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60614: Service unavailable: leader transfer in progress
I20240503 01:59:15.060956 5516 raft_consensus.cc:979] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: : Instructing follower 0fb6a1bb16214e3c9ec6acd1ab111358 to start an election
I20240503 01:59:15.061038 5516 raft_consensus.cc:1067] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Signalling peer 0fb6a1bb16214e3c9ec6acd1ab111358 to start an election
I20240503 01:59:15.061600 5318 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e504950788d7408a83d4b7299b78518c"
dest_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358"
from {username='slave'} at 127.0.99.129:52693
I20240503 01:59:15.061779 5318 raft_consensus.cc:482] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:15.061906 5318 raft_consensus.cc:3037] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.063099 5318 raft_consensus.cc:504] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
W20240503 01:59:15.065492 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60614: Service unavailable: leader transfer in progress
I20240503 01:59:15.065654 3677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504950788d7408a83d4b7299b78518c" candidate_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" candidate_term: 2 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: true dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:15.065757 3677 raft_consensus.cc:3032] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:15.065800 3677 raft_consensus.cc:729] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:15.065935 3677 consensus_queue.cc:260] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 1.151, Last appended by leader: 151, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:15.066104 3677 raft_consensus.cc:3037] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.067150 3677 raft_consensus.cc:2454] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb6a1bb16214e3c9ec6acd1ab111358 in term 2.
I20240503 01:59:15.069248 5316 raft_consensus.cc:1226] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 2 FOLLOWER]: Rejecting Update request from peer 2820cbd55ee1470880e35a8ab671563b for earlier term 1. Current term is 2. Ops: []
I20240503 01:59:15.069407 5252 leader_election.cc:302] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb6a1bb16214e3c9ec6acd1ab111358, 2820cbd55ee1470880e35a8ab671563b; no voters:
I20240503 01:59:15.069667 6025 raft_consensus.cc:2781] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:15.069834 6025 raft_consensus.cc:686] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 2 LEADER]: Becoming Leader. State: Replica: 0fb6a1bb16214e3c9ec6acd1ab111358, State: Running, Role: LEADER
I20240503 01:59:15.070086 5318 leader_election.cc:288] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [CANDIDATE]: Term 2 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461), c6f646ec3edd4e298f16122028bb0ef7 (127.0.99.133:41793)
I20240503 01:59:15.070127 6025 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.151, Last appended by leader: 151, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
W20240503 01:59:15.073758 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60614: Illegal state: replica 2820cbd55ee1470880e35a8ab671563b is not leader of this config: current role FOLLOWER
I20240503 01:59:15.074168 5186 raft_consensus.cc:1203] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.121->[2.122-2.168] Dedup: 2.147->[2.148-2.168]
I20240503 01:59:15.076279 3531 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 reported cstate change: term changed from 1 to 2, leader changed from 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) to 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:15.080255 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504950788d7408a83d4b7299b78518c" candidate_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" candidate_term: 2 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: true dest_uuid: "c6f646ec3edd4e298f16122028bb0ef7"
I20240503 01:59:15.080403 5054 raft_consensus.cc:3037] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.081517 5054 raft_consensus.cc:2454] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb6a1bb16214e3c9ec6acd1ab111358 in term 2.
W20240503 01:59:15.087493 3636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60614: Illegal state: replica 2820cbd55ee1470880e35a8ab671563b is not leader of this config: current role FOLLOWER
W20240503 01:59:15.118891 5014 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49654: Illegal state: replica c6f646ec3edd4e298f16122028bb0ef7 is not leader of this config: current role FOLLOWER
I20240503 01:59:15.139484 5433 raft_consensus.cc:1050] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 7a367744629a43edb0df92463a498258 to VOTER
I20240503 01:59:15.139866 5433 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 168, Committed index: 168, Last appended: 2.169, Last appended by leader: 92, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:15.141105 4920 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 2 index: 169. Preceding OpId from leader: term: 2 index: 170. (index mismatch)
I20240503 01:59:15.141690 6041 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 168, Time since last communication: 0.000s
I20240503 01:59:15.141813 5186 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 2 index: 169. Preceding OpId from leader: term: 2 index: 170. (index mismatch)
I20240503 01:59:15.143654 6041 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Committing config change with OpId 2.170: config changed from index 122 to 170, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.144662 4920 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Committing config change with OpId 2.170: config changed from index 122 to 170, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.146907 3531 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 122 to 170, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:15.149256 5433 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 168, Time since last communication: 0.000s
I20240503 01:59:15.152448 5186 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Committing config change with OpId 2.170: config changed from index 122 to 170, 7a367744629a43edb0df92463a498258 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.181120 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Refusing update from remote peer 0fb6a1bb16214e3c9ec6acd1ab111358: Log matching property violated. Preceding OpId in replica: term: 1 index: 151. Preceding OpId from leader: term: 2 index: 153. (index mismatch)
I20240503 01:59:15.181605 6078 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I20240503 01:59:15.181994 3678 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer 0fb6a1bb16214e3c9ec6acd1ab111358: Log matching property violated. Preceding OpId in replica: term: 1 index: 151. Preceding OpId from leader: term: 2 index: 153. (index mismatch)
I20240503 01:59:15.183391 6078 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I20240503 01:59:15.188030 6016 mvcc.cc:205] Tried to move back new op lower bound from 7023417570016124928 to 7023417569569144832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417570016124928 or (T in {7023417570016124928})}]
I20240503 01:59:15.192705 5650 mvcc.cc:205] Tried to move back new op lower bound from 7023417570016124928 to 7023417569569144832. Current Snapshot: MvccSnapshot[applied={T|T < 7023417570016124928}]
I20240503 01:59:15.244949 5315 consensus_queue.cc:237] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 2.155, Last appended by leader: 151, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } }
I20240503 01:59:15.246330 5054 raft_consensus.cc:1261] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Refusing update from remote peer 0fb6a1bb16214e3c9ec6acd1ab111358: Log matching property violated. Preceding OpId in replica: term: 2 index: 155. Preceding OpId from leader: term: 2 index: 156. (index mismatch)
I20240503 01:59:15.247184 6078 consensus_queue.cc:1035] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240503 01:59:15.248991 6078 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 2 LEADER]: Committing config change with OpId 2.156: config changed from index 140 to 156, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:15.250573 3522 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e504950788d7408a83d4b7299b78518c with cas_config_opid_index 140: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:15.252794 5054 raft_consensus.cc:2932] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 [term 2 FOLLOWER]: Committing config change with OpId 2.156: config changed from index 140 to 156, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:15.255271 3534 catalog_manager.cc:5484] T e504950788d7408a83d4b7299b78518c P c6f646ec3edd4e298f16122028bb0ef7 reported cstate change: config changed from index 140 to 156, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "c6f646ec3edd4e298f16122028bb0ef7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41793 } attrs { promote: false } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false } } }
I20240503 01:59:15.261713 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet e504950788d7408a83d4b7299b78518c with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 156) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:15.288956 6089 tablet_replica.cc:330] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:15.291664 6089 raft_consensus.cc:2229] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:15.291972 6089 raft_consensus.cc:2258] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:15.294855 6089 ts_tablet_manager.cc:1875] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:15.309279 6089 ts_tablet_manager.cc:1888] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.155
I20240503 01:59:15.309393 6089 log.cc:1196] T e504950788d7408a83d4b7299b78518c P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e504950788d7408a83d4b7299b78518c
I20240503 01:59:15.310163 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet e504950788d7408a83d4b7299b78518c (table rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]) successfully deleted
I20240503 01:59:15.360785 3938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46828
I20240503 01:59:15.360931 3938 raft_consensus.cc:595] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Received request to transfer leadership
I20240503 01:59:15.362573 4836 raft_consensus.cc:979] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: : Instructing follower 7a367744629a43edb0df92463a498258 to start an election
I20240503 01:59:15.362686 4836 raft_consensus.cc:1067] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Signalling peer 7a367744629a43edb0df92463a498258 to start an election
W20240503 01:59:15.364519 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46576: Service unavailable: leader transfer in progress
I20240503 01:59:15.366257 5185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a"
dest_uuid: "7a367744629a43edb0df92463a498258"
from {username='slave'} at 127.0.99.131:53945
I20240503 01:59:15.366397 5185 raft_consensus.cc:482] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:15.366462 5185 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:15.367587 5185 raft_consensus.cc:504] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
W20240503 01:59:15.370474 3900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46576: Service unavailable: leader transfer in progress
I20240503 01:59:15.371690 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "7a367744629a43edb0df92463a498258" candidate_term: 3 candidate_status { last_received { term: 2 index: 181 } } ignore_live_leader: true dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:15.371802 3938 raft_consensus.cc:3032] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 01:59:15.371840 3938 raft_consensus.cc:729] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:15.371879 5185 leader_election.cc:288] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [CANDIDATE]: Term 3 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705), 7b5c836ac65b442e808f46fb2d267d2c (127.0.99.132:35901)
I20240503 01:59:15.371976 3938 consensus_queue.cc:260] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 2.181, Last appended by leader: 181, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:15.372121 3938 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:15.373148 3938 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a367744629a43edb0df92463a498258 in term 3.
I20240503 01:59:15.376245 5121 leader_election.cc:302] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a367744629a43edb0df92463a498258, c8595f9ae8f644d6901791810d7b970e; no voters:
I20240503 01:59:15.376510 5822 raft_consensus.cc:2781] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 01:59:15.376590 5822 raft_consensus.cc:686] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 3 LEADER]: Becoming Leader. State: Replica: 7a367744629a43edb0df92463a498258, State: Running, Role: LEADER
I20240503 01:59:15.376737 5822 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 2.181, Last appended by leader: 181, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:15.379109 3531 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 reported cstate change: term changed from 2 to 3, leader changed from c8595f9ae8f644d6901791810d7b970e (127.0.99.131) to 7a367744629a43edb0df92463a498258 (127.0.99.134). New cstate: current_term: 3 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 01:59:15.380383 3899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46576: Illegal state: replica c8595f9ae8f644d6901791810d7b970e is not leader of this config: current role FOLLOWER
I20240503 01:59:15.385960 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7cf094bb2546bb800180d3a530f46a" candidate_uuid: "7a367744629a43edb0df92463a498258" candidate_term: 3 candidate_status { last_received { term: 2 index: 181 } } ignore_live_leader: true dest_uuid: "7b5c836ac65b442e808f46fb2d267d2c"
I20240503 01:59:15.386099 4920 raft_consensus.cc:3037] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:15.387216 4920 raft_consensus.cc:2454] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a367744629a43edb0df92463a498258 in term 3.
I20240503 01:59:15.412104 4920 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 3 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 181. Preceding OpId from leader: term: 3 index: 183. (index mismatch)
I20240503 01:59:15.412371 3938 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 3 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 181. Preceding OpId from leader: term: 3 index: 183. (index mismatch)
I20240503 01:59:15.412539 5636 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:15.412761 5822 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240503 01:59:15.426695 5186 consensus_queue.cc:237] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 3.184, Last appended by leader: 181, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } }
I20240503 01:59:15.428329 4919 raft_consensus.cc:1261] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 3 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 3 index: 184. Preceding OpId from leader: term: 3 index: 185. (index mismatch)
I20240503 01:59:15.428891 5636 consensus_queue.cc:1035] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 183, Time since last communication: 0.000s
I20240503 01:59:15.430953 5636 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7a367744629a43edb0df92463a498258 [term 3 LEADER]: Committing config change with OpId 3.185: config changed from index 170 to 185, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.432389 3522 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a7cf094bb2546bb800180d3a530f46a with cas_config_opid_index 170: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:15.434937 4919 raft_consensus.cc:2932] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c [term 3 FOLLOWER]: Committing config change with OpId 3.185: config changed from index 170 to 185, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.437654 3534 catalog_manager.cc:5484] T 6a7cf094bb2546bb800180d3a530f46a P 7b5c836ac65b442e808f46fb2d267d2c reported cstate change: config changed from index 170 to 185, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 3 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "7b5c836ac65b442e808f46fb2d267d2c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35901 } attrs { promote: false } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } }
I20240503 01:59:15.441941 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7cf094bb2546bb800180d3a530f46a with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 185) from {username='slave'} at 127.0.0.1:46500
I20240503 01:59:15.442313 6117 tablet_replica.cc:330] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
I20240503 01:59:15.442504 6117 raft_consensus.cc:2229] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:15.442685 6117 pending_rounds.cc:62] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Trying to abort 1 pending ops.
I20240503 01:59:15.442883 6117 pending_rounds.cc:68] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Aborting op as it isn't in flight: id { term: 3 index: 184 } timestamp: 7023417571019345920 op_type: WRITE_OP write_request { tablet_id: "6a7cf094bb2546bb800180d3a530f46a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\n\205\306\037\223\201\221>\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""33b998d4bbc1834e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417570999808000 authz_token { token_data: "\010\232\214\321\261\006\"3\n\005slave\022*\n b1fafe774392410fa81d5ffd0a54b8e8\020\001\030\001 \001(\001" signature: "<redacted>""\304\360\226\372*\230c\025\021\237\027\257\264\233\243\262\023r\3038\3216i\376\301\032\346C\001+W<\245\270\245\225\306\316x\tE\345\332Xa\331[\255q\327\032\001|\316\332\260\265.G\245\"yU\224" signing_key_seq_num: 0 } } request_id { client_id: "429534f175fe4b018bf5f2b2f1306065" seq_no: 494 first_incomplete_seq_no: 494 attempt_no: 0 }
I20240503 01:59:15.443387 6117 raft_consensus.cc:2258] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:15.444155 6117 ts_tablet_manager.cc:1875] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:15.446836 6117 ts_tablet_manager.cc:1888] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.184
I20240503 01:59:15.446960 6117 log.cc:1196] T 6a7cf094bb2546bb800180d3a530f46a P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:15.447916 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet 6a7cf094bb2546bb800180d3a530f46a (table rebalance_test_table_0 [id=b1fafe774392410fa81d5ffd0a54b8e8]) successfully deleted
I20240503 01:59:15.636878 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.647797 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:15.650993 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.674502 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.684537 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:15.703161 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:15.735520 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:16.032895 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:16.039626 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.040393 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:16.053063 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.101233 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:16.113994 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.120986 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:16.714324 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:16.729154 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:16.762197 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:16.772794 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.784407 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.838703 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.841215 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:17.177610 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.197012 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.205557 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:17.252224 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.259578 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.262364 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:17.274328 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.458751 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.461365 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.463073 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:17.464139 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:17.466992 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.468127 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:17.473696 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d88debf910664f928bbd978e084af0e0 | 127.0.99.190:33221 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39243 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0fb6a1bb16214e3c9ec6acd1ab111358 | 127.0.99.135:42189 | HEALTHY | <none> | 1 | 0
2820cbd55ee1470880e35a8ab671563b | 127.0.99.129:43461 | HEALTHY | <none> | 4 | 0
7a367744629a43edb0df92463a498258 | 127.0.99.134:42343 | HEALTHY | <none> | 2 | 0
7b5c836ac65b442e808f46fb2d267d2c | 127.0.99.132:35901 | HEALTHY | <none> | 2 | 0
c6f646ec3edd4e298f16122028bb0ef7 | 127.0.99.133:41793 | HEALTHY | <none> | 1 | 0
c8595f9ae8f644d6901791810d7b970e | 127.0.99.131:36705 | HEALTHY | <none> | 4 | 0
f929df19fdfc4df1986e990c60056733 | 127.0.99.130:35911 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43461
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:35911
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:36705
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35901
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41793
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:42343
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:42189
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43461
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:35911
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:36705
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35901
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41793
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:42343
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:42189
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39243 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:17.509155 398 log_verifier.cc:126] Checking tablet 02a421366d744afd99d7779644e9e41e
I20240503 01:59:17.555922 398 log_verifier.cc:177] Verified matching terms for 222 ops in tablet 02a421366d744afd99d7779644e9e41e
I20240503 01:59:17.556026 398 log_verifier.cc:126] Checking tablet 1c6b5cb588234440807eee9875c334c1
I20240503 01:59:17.604283 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet 1c6b5cb588234440807eee9875c334c1
I20240503 01:59:17.604449 398 log_verifier.cc:126] Checking tablet 287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:17.660004 398 log_verifier.cc:177] Verified matching terms for 227 ops in tablet 287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:17.660105 398 log_verifier.cc:126] Checking tablet 4baa1b72653c4a0ba130008b06bc5897
I20240503 01:59:17.711112 398 log_verifier.cc:177] Verified matching terms for 232 ops in tablet 4baa1b72653c4a0ba130008b06bc5897
I20240503 01:59:17.711247 398 log_verifier.cc:126] Checking tablet 4d2ab9fbe93a44adbc34f306e8d61c27
I20240503 01:59:17.752725 398 log_verifier.cc:177] Verified matching terms for 203 ops in tablet 4d2ab9fbe93a44adbc34f306e8d61c27
I20240503 01:59:17.752835 398 log_verifier.cc:126] Checking tablet 56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:17.802559 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet 56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:17.802754 398 log_verifier.cc:126] Checking tablet 5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:17.854557 398 log_verifier.cc:177] Verified matching terms for 217 ops in tablet 5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:17.854660 398 log_verifier.cc:126] Checking tablet 6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:17.902284 398 log_verifier.cc:177] Verified matching terms for 228 ops in tablet 6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:17.902402 398 log_verifier.cc:126] Checking tablet 6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:17.954417 398 log_verifier.cc:177] Verified matching terms for 254 ops in tablet 6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:17.954533 398 log_verifier.cc:126] Checking tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:18.003468 398 log_verifier.cc:177] Verified matching terms for 229 ops in tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:18.003633 398 log_verifier.cc:126] Checking tablet b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:18.052839 398 log_verifier.cc:177] Verified matching terms for 197 ops in tablet b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:18.052971 398 log_verifier.cc:126] Checking tablet cbba240087984f5bac2370d46a09a436
I20240503 01:59:18.100581 398 log_verifier.cc:177] Verified matching terms for 232 ops in tablet cbba240087984f5bac2370d46a09a436
I20240503 01:59:18.100690 398 log_verifier.cc:126] Checking tablet e504950788d7408a83d4b7299b78518c
I20240503 01:59:18.145044 398 log_verifier.cc:177] Verified matching terms for 216 ops in tablet e504950788d7408a83d4b7299b78518c
I20240503 01:59:18.145167 398 log_verifier.cc:126] Checking tablet e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:18.191666 398 log_verifier.cc:177] Verified matching terms for 214 ops in tablet e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:18.191802 398 log_verifier.cc:126] Checking tablet ea7e73129def488bbf9ed086e424968e
I20240503 01:59:18.206411 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ea7e73129def488bbf9ed086e424968e
I20240503 01:59:18.206475 398 log_verifier.cc:126] Checking tablet f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:18.256572 398 log_verifier.cc:177] Verified matching terms for 254 ops in tablet f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:18.258702 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--builtin_ntp_servers=127.0.99.148:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:18.370044 6339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.370275 6339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.370342 6339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.373754 6339 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:18.373854 6339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.373955 6339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:18.378086 6339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33221
--never_fsync=true
--heap_profile_path=/tmp/kudu.6339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.379115 6339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.380396 6339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.389545 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.390553 6347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.391455 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.392596 6339 server_base.cc:1034] running on GCE node
I20240503 01:59:18.393045 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.393718 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.394891 6339 hybrid_clock.cc:648] HybridClock initialized: now 1714701558394873 us; error 30 us; skew 500 ppm
I20240503 01:59:18.396960 6339 webserver.cc:460] Webserver started at http://127.0.99.136:39379/ using document root <none> and password file <none>
I20240503 01:59:18.397508 6339 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.397563 6339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.397718 6339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.399389 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "11863510d4f04be8becdb64e1dff19cc"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-pvfz"
I20240503 01:59:18.399884 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "11863510d4f04be8becdb64e1dff19cc"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-pvfz"
I20240503 01:59:18.403585 6339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:18.405628 6353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.406625 6339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:18.406728 6339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "11863510d4f04be8becdb64e1dff19cc"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-pvfz"
I20240503 01:59:18.406841 6339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.418093 6339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.418800 6339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.418964 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.419520 6339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.420454 6339 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.420508 6339 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.420555 6339 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.420581 6339 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.428758 6339 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:39581
I20240503 01:59:18.428772 6466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:39581 every 8 connection(s)
I20240503 01:59:18.429774 6339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:18.434875 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 6339
I20240503 01:59:18.434990 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:18.439055 6467 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33221
I20240503 01:59:18.439293 6467 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.439836 6467 heartbeater.cc:511] Master 127.0.99.190:33221 requested a full tablet report, sending...
I20240503 01:59:18.440707 3531 ts_manager.cc:194] Registered new tserver with Master: 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581)
I20240503 01:59:18.441267 3531 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34913
I20240503 01:59:18.625730 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.626808 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.629199 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.629777 6401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.631613 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.631695 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.653720 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.668520 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.752710 3657 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.754205 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.754865 6401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:18.761086 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.765453 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.765643 3920 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:18.769235 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:18.792727 3788 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:18.824383 5186 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 60, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:18.826027 3809 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240503 01:59:18.826519 6257 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240503 01:59:18.828449 6563 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 63 to 228, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
I20240503 01:59:18.828895 3809 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 63 to 228, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
W20240503 01:59:18.831024 5123 consensus_peers.cc:485] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 -> Peer 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581): Couldn't send request to peer 11863510d4f04be8becdb64e1dff19cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 287bd1cfeb1e4e709d40fb3368aaca77. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.831509 3536 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 reported cstate change: config changed from index 63 to 228, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:18.839303 3938 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:18.840916 5316 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 01:59:18.841411 6576 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 01:59:18.843339 5452 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 81 to 235, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
I20240503 01:59:18.843878 5316 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 81 to 235, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
W20240503 01:59:18.845110 3876 consensus_peers.cc:485] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e -> Peer 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581): Couldn't send request to peer 11863510d4f04be8becdb64e1dff19cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56cd6bd86d2a4391947f7b7193b5e420. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.846906 3536 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 81 to 235, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:18.854214 3938 consensus_queue.cc:237] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:18.855477 3809 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 1 index: 198. (index mismatch)
W20240503 01:59:18.855695 3876 consensus_peers.cc:485] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e -> Peer 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581): Couldn't send request to peer 11863510d4f04be8becdb64e1dff19cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1aa056182cf435d8a6f675aa2b42129. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.855859 6576 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I20240503 01:59:18.857367 5452 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.198: config changed from index -1 to 198, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
I20240503 01:59:18.857745 3809 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.198: config changed from index -1 to 198, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
I20240503 01:59:18.860137 3529 catalog_manager.cc:5484] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index -1 to 198, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:18.867712 3678 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:18.869256 5186 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 255. (index mismatch)
I20240503 01:59:18.869727 6592 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20240503 01:59:18.871398 5384 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.255: config changed from index 70 to 255, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
W20240503 01:59:18.872576 3614 consensus_peers.cc:485] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b -> Peer 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581): Couldn't send request to peer 11863510d4f04be8becdb64e1dff19cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8d39433e68b4bd7b14b42e850f42ae6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.873113 5186 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.255: config changed from index 70 to 255, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } } }
I20240503 01:59:18.873824 3531 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 70 to 255, NON_VOTER 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:19.251255 6635 ts_tablet_manager.cc:902] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:19.252171 6635 tablet_copy_client.cc:308] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:19.252755 6640 ts_tablet_manager.cc:902] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Initiating tablet copy from peer 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461)
I20240503 01:59:19.253396 6640 tablet_copy_client.cc:308] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43461
I20240503 01:59:19.256860 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet b1aa056182cf435d8a6f675aa2b42129 from peer 11863510d4f04be8becdb64e1dff19cc ({username='slave'} at 127.0.99.136:42177)
I20240503 01:59:19.256966 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet b1aa056182cf435d8a6f675aa2b42129 from peer 11863510d4f04be8becdb64e1dff19cc at {username='slave'} at 127.0.99.136:42177: session id = 11863510d4f04be8becdb64e1dff19cc-b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:19.256959 3697 tablet_copy_service.cc:140] P 2820cbd55ee1470880e35a8ab671563b: Received BeginTabletCopySession request for tablet f8d39433e68b4bd7b14b42e850f42ae6 from peer 11863510d4f04be8becdb64e1dff19cc ({username='slave'} at 127.0.99.136:34751)
I20240503 01:59:19.257071 3697 tablet_copy_service.cc:161] P 2820cbd55ee1470880e35a8ab671563b: Beginning new tablet copy session on tablet f8d39433e68b4bd7b14b42e850f42ae6 from peer 11863510d4f04be8becdb64e1dff19cc at {username='slave'} at 127.0.99.136:34751: session id = 11863510d4f04be8becdb64e1dff19cc-f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:19.257884 3697 tablet_copy_source_session.cc:205] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.257884 3960 tablet_copy_source_session.cc:205] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.261267 6635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aa056182cf435d8a6f675aa2b42129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.261592 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d39433e68b4bd7b14b42e850f42ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.267263 6635 tablet_copy_client.cc:783] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.267278 6640 tablet_copy_client.cc:783] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.267637 6640 tablet_copy_client.cc:647] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.267660 6635 tablet_copy_client.cc:647] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.270248 6635 tablet_copy_client.cc:523] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.270757 6640 tablet_copy_client.cc:523] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.272186 6635 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap starting.
I20240503 01:59:19.272186 6640 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap starting.
I20240503 01:59:19.289333 6635 log.cc:824] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:19.298223 6649 ts_tablet_manager.cc:902] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Initiating tablet copy from peer 7a367744629a43edb0df92463a498258 (127.0.99.134:42343)
I20240503 01:59:19.298754 6649 tablet_copy_client.cc:308] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42343
I20240503 01:59:19.303059 5207 tablet_copy_service.cc:140] P 7a367744629a43edb0df92463a498258: Received BeginTabletCopySession request for tablet 287bd1cfeb1e4e709d40fb3368aaca77 from peer 11863510d4f04be8becdb64e1dff19cc ({username='slave'} at 127.0.99.136:35091)
I20240503 01:59:19.303215 5207 tablet_copy_service.cc:161] P 7a367744629a43edb0df92463a498258: Beginning new tablet copy session on tablet 287bd1cfeb1e4e709d40fb3368aaca77 from peer 11863510d4f04be8becdb64e1dff19cc at {username='slave'} at 127.0.99.136:35091: session id = 11863510d4f04be8becdb64e1dff19cc-287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:19.304638 5207 tablet_copy_source_session.cc:205] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.305714 6649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287bd1cfeb1e4e709d40fb3368aaca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.307830 6649 tablet_copy_client.cc:783] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.307966 6649 tablet_copy_client.cc:647] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.310178 6649 tablet_copy_client.cc:523] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.311381 6649 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap starting.
I20240503 01:59:19.360196 6675 ts_tablet_manager.cc:902] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Initiating tablet copy from peer c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705)
I20240503 01:59:19.360710 6675 tablet_copy_client.cc:308] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36705
I20240503 01:59:19.361146 3960 tablet_copy_service.cc:140] P c8595f9ae8f644d6901791810d7b970e: Received BeginTabletCopySession request for tablet 56cd6bd86d2a4391947f7b7193b5e420 from peer 11863510d4f04be8becdb64e1dff19cc ({username='slave'} at 127.0.99.136:42177)
I20240503 01:59:19.361263 3960 tablet_copy_service.cc:161] P c8595f9ae8f644d6901791810d7b970e: Beginning new tablet copy session on tablet 56cd6bd86d2a4391947f7b7193b5e420 from peer 11863510d4f04be8becdb64e1dff19cc at {username='slave'} at 127.0.99.136:42177: session id = 11863510d4f04be8becdb64e1dff19cc-56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:19.362174 3960 tablet_copy_source_session.cc:205] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.362987 6675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cd6bd86d2a4391947f7b7193b5e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.365171 6675 tablet_copy_client.cc:783] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.365329 6675 tablet_copy_client.cc:647] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.367168 6675 tablet_copy_client.cc:523] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:19.368496 6675 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap starting.
I20240503 01:59:19.433820 6635 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=198 overwritten=0 applied=198 ignored=0} inserts{seen=870 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.434250 6635 tablet_bootstrap.cc:492] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap complete.
I20240503 01:59:19.434609 6635 ts_tablet_manager.cc:1367] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Time spent bootstrapping tablet: real 0.163s user 0.143s sys 0.013s
I20240503 01:59:19.437275 6635 raft_consensus.cc:348] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.437536 6635 raft_consensus.cc:729] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11863510d4f04be8becdb64e1dff19cc, State: Initialized, Role: LEARNER
I20240503 01:59:19.437894 6635 consensus_queue.cc:260] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 198, Last appended: 1.198, Last appended by leader: 198, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.438813 6635 ts_tablet_manager.cc:1398] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:19.439210 6467 heartbeater.cc:503] Master 127.0.99.190:33221 was elected leader, sending a full tablet report...
I20240503 01:59:19.439639 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 11863510d4f04be8becdb64e1dff19cc-b1aa056182cf435d8a6f675aa2b42129 received from {username='slave'} at 127.0.99.136:42177
I20240503 01:59:19.439728 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 11863510d4f04be8becdb64e1dff19cc-b1aa056182cf435d8a6f675aa2b42129 on tablet b1aa056182cf435d8a6f675aa2b42129 with peer 11863510d4f04be8becdb64e1dff19cc
I20240503 01:59:19.488777 6640 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=255 overwritten=0 applied=255 ignored=0} inserts{seen=937 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.489126 6640 tablet_bootstrap.cc:492] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap complete.
I20240503 01:59:19.489356 6640 ts_tablet_manager.cc:1367] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Time spent bootstrapping tablet: real 0.217s user 0.180s sys 0.006s
I20240503 01:59:19.490013 6640 raft_consensus.cc:348] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.490141 6640 raft_consensus.cc:729] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11863510d4f04be8becdb64e1dff19cc, State: Initialized, Role: LEARNER
I20240503 01:59:19.490255 6640 consensus_queue.cc:260] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 255, Last appended: 1.255, Last appended by leader: 255, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.490556 6640 ts_tablet_manager.cc:1398] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:19.491385 3697 tablet_copy_service.cc:342] P 2820cbd55ee1470880e35a8ab671563b: Request end of tablet copy session 11863510d4f04be8becdb64e1dff19cc-f8d39433e68b4bd7b14b42e850f42ae6 received from {username='slave'} at 127.0.99.136:34751
I20240503 01:59:19.491469 3697 tablet_copy_service.cc:433] P 2820cbd55ee1470880e35a8ab671563b: ending tablet copy session 11863510d4f04be8becdb64e1dff19cc-f8d39433e68b4bd7b14b42e850f42ae6 on tablet f8d39433e68b4bd7b14b42e850f42ae6 with peer 11863510d4f04be8becdb64e1dff19cc
I20240503 01:59:19.510403 6649 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=228 overwritten=0 applied=228 ignored=0} inserts{seen=896 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.510706 6649 tablet_bootstrap.cc:492] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap complete.
I20240503 01:59:19.510895 6649 ts_tablet_manager.cc:1367] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Time spent bootstrapping tablet: real 0.200s user 0.152s sys 0.017s
I20240503 01:59:19.511334 6649 raft_consensus.cc:348] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.511438 6649 raft_consensus.cc:729] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 11863510d4f04be8becdb64e1dff19cc, State: Initialized, Role: LEARNER
I20240503 01:59:19.511529 6649 consensus_queue.cc:260] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 228, Last appended: 2.228, Last appended by leader: 228, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.511852 6649 ts_tablet_manager.cc:1398] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 01:59:19.512512 5207 tablet_copy_service.cc:342] P 7a367744629a43edb0df92463a498258: Request end of tablet copy session 11863510d4f04be8becdb64e1dff19cc-287bd1cfeb1e4e709d40fb3368aaca77 received from {username='slave'} at 127.0.99.136:35091
I20240503 01:59:19.512622 5207 tablet_copy_service.cc:433] P 7a367744629a43edb0df92463a498258: ending tablet copy session 11863510d4f04be8becdb64e1dff19cc-287bd1cfeb1e4e709d40fb3368aaca77 on tablet 287bd1cfeb1e4e709d40fb3368aaca77 with peer 11863510d4f04be8becdb64e1dff19cc
I20240503 01:59:19.537755 6675 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=235 overwritten=0 applied=235 ignored=0} inserts{seen=863 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:19.538074 6675 tablet_bootstrap.cc:492] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Bootstrap complete.
I20240503 01:59:19.538267 6675 ts_tablet_manager.cc:1367] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Time spent bootstrapping tablet: real 0.170s user 0.155s sys 0.008s
I20240503 01:59:19.538933 6675 raft_consensus.cc:348] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.539045 6675 raft_consensus.cc:729] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11863510d4f04be8becdb64e1dff19cc, State: Initialized, Role: LEARNER
I20240503 01:59:19.539147 6675 consensus_queue.cc:260] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 235, Last appended: 1.235, Last appended by leader: 235, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: true } }
I20240503 01:59:19.539403 6675 ts_tablet_manager.cc:1398] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:19.540058 3960 tablet_copy_service.cc:342] P c8595f9ae8f644d6901791810d7b970e: Request end of tablet copy session 11863510d4f04be8becdb64e1dff19cc-56cd6bd86d2a4391947f7b7193b5e420 received from {username='slave'} at 127.0.99.136:42177
I20240503 01:59:19.540158 3960 tablet_copy_service.cc:433] P c8595f9ae8f644d6901791810d7b970e: ending tablet copy session 11863510d4f04be8becdb64e1dff19cc-56cd6bd86d2a4391947f7b7193b5e420 on tablet 56cd6bd86d2a4391947f7b7193b5e420 with peer 11863510d4f04be8becdb64e1dff19cc
I20240503 01:59:19.741475 6421 raft_consensus.cc:1203] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.197->[1.198-1.198] Dedup: 1.198->[]
I20240503 01:59:19.782953 6421 raft_consensus.cc:1203] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.227->[2.228-2.228] Dedup: 2.228->[]
I20240503 01:59:19.849056 6421 raft_consensus.cc:1203] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.254->[1.255-1.255] Dedup: 1.255->[]
I20240503 01:59:19.861374 6421 raft_consensus.cc:1203] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.234->[1.235-1.235] Dedup: 1.235->[]
I20240503 01:59:20.307880 6704 raft_consensus.cc:1050] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 11863510d4f04be8becdb64e1dff19cc to VOTER
I20240503 01:59:20.308359 6704 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.309765 5316 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 01:59:20.310217 6576 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 01:59:20.310418 6703 raft_consensus.cc:1050] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: attempting to promote NON_VOTER 11863510d4f04be8becdb64e1dff19cc to VOTER
I20240503 01:59:20.310444 6421 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 01:59:20.310786 6703 consensus_queue.cc:237] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.198, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.310886 6576 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 01:59:20.312124 6420 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240503 01:59:20.312400 6703 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 235 to 236, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.312773 5316 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.314411 3809 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240503 01:59:20.314394 6773 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240503 01:59:20.315062 6775 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240503 01:59:20.315469 3529 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 reported cstate change: config changed from index 235 to 236, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.315768 6420 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "0fb6a1bb16214e3c9ec6acd1ab111358" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42189 } attrs { promote: false replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.317771 6576 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.199: config changed from index 198 to 199, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.320667 3809 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 198 to 199, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.322681 6421 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 198 to 199, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.323513 3536 catalog_manager.cc:5484] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index 198 to 199, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.326998 3938 consensus_queue.cc:237] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.328172 6421 raft_consensus.cc:1261] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Refusing update from remote peer c8595f9ae8f644d6901791810d7b970e: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240503 01:59:20.328491 6775 consensus_queue.cc:1035] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 01:59:20.330121 6773 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 236 to 237, VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.330771 6421 raft_consensus.cc:2932] T 56cd6bd86d2a4391947f7b7193b5e420 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.333585 3521 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56cd6bd86d2a4391947f7b7193b5e420 with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.335410 3536 catalog_manager.cc:5484] T 56cd6bd86d2a4391947f7b7193b5e420 P c8595f9ae8f644d6901791810d7b970e reported cstate change: config changed from index 236 to 237, VOTER 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "c8595f9ae8f644d6901791810d7b970e" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.338537 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cd6bd86d2a4391947f7b7193b5e420 with delete_type TABLET_DATA_TOMBSTONED (TS 0fb6a1bb16214e3c9ec6acd1ab111358 not found in new config with opid_index 237) from {username='slave'} at 127.0.0.1:40528
I20240503 01:59:20.339118 6782 tablet_replica.cc:330] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: stopping tablet replica
I20240503 01:59:20.339470 6782 raft_consensus.cc:2229] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.339779 6782 raft_consensus.cc:2258] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.341828 6782 ts_tablet_manager.cc:1875] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.343369 6727 raft_consensus.cc:1050] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: attempting to promote NON_VOTER 11863510d4f04be8becdb64e1dff19cc to VOTER
I20240503 01:59:20.343930 6727 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.344100 6782 ts_tablet_manager.cc:1888] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.236
I20240503 01:59:20.344220 6782 log.cc:1196] T 56cd6bd86d2a4391947f7b7193b5e420 P 0fb6a1bb16214e3c9ec6acd1ab111358: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:20.345528 6421 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 LEARNER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240503 01:59:20.345739 5186 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Refusing update from remote peer 2820cbd55ee1470880e35a8ab671563b: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20240503 01:59:20.345798 3522 catalog_manager.cc:4830] TS 0fb6a1bb16214e3c9ec6acd1ab111358 (127.0.99.135:42189): tablet 56cd6bd86d2a4391947f7b7193b5e420 (table rebalance_test_table_3 [id=46e61805ee29430c9f6d81a5f398968b]) successfully deleted
I20240503 01:59:20.345973 5384 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20240503 01:59:20.346241 6726 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20240503 01:59:20.348325 6257 raft_consensus.cc:1050] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258: attempting to promote NON_VOTER 11863510d4f04be8becdb64e1dff19cc to VOTER
I20240503 01:59:20.348357 6727 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Committing config change with OpId 1.256: config changed from index 255 to 256, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.348757 6257 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 60, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.348780 6421 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 255 to 256, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.350476 3809 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20240503 01:59:20.350778 6421 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEARNER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20240503 01:59:20.351152 3536 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b reported cstate change: config changed from index 255 to 256, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2820cbd55ee1470880e35a8ab671563b" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.351408 6257 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20240503 01:59:20.351599 5186 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 255 to 256, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.351796 6257 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20240503 01:59:20.354279 6257 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.354630 3809 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.355683 6421 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.358565 3532 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index 228 to 229, 11863510d4f04be8becdb64e1dff19cc (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.366000 5186 consensus_queue.cc:237] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 60, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.367074 6421 raft_consensus.cc:1261] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Refusing update from remote peer 7a367744629a43edb0df92463a498258: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20240503 01:59:20.367677 6788 consensus_queue.cc:1035] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20240503 01:59:20.369536 6786 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.369971 6421 raft_consensus.cc:2932] T 287bd1cfeb1e4e709d40fb3368aaca77 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:20.371877 3522 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 287bd1cfeb1e4e709d40fb3368aaca77 with cas_config_opid_index 229: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:20.372247 3532 catalog_manager.cc:5484] T 287bd1cfeb1e4e709d40fb3368aaca77 P 7a367744629a43edb0df92463a498258 reported cstate change: config changed from index 229 to 230, VOTER f929df19fdfc4df1986e990c60056733 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "7a367744629a43edb0df92463a498258" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:20.374487 3788 tablet_service.cc:1514] Processing DeleteTablet for tablet 287bd1cfeb1e4e709d40fb3368aaca77 with delete_type TABLET_DATA_TOMBSTONED (TS f929df19fdfc4df1986e990c60056733 not found in new config with opid_index 230) from {username='slave'} at 127.0.0.1:35928
I20240503 01:59:20.374871 6792 tablet_replica.cc:330] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: stopping tablet replica
I20240503 01:59:20.375011 6792 raft_consensus.cc:2229] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:20.375186 6792 raft_consensus.cc:2258] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:20.376636 6792 ts_tablet_manager.cc:1875] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:20.378768 6792 ts_tablet_manager.cc:1888] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.229
I20240503 01:59:20.378854 6792 log.cc:1196] T 287bd1cfeb1e4e709d40fb3368aaca77 P f929df19fdfc4df1986e990c60056733: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:20.379374 3520 catalog_manager.cc:4830] TS f929df19fdfc4df1986e990c60056733 (127.0.99.130:35911): tablet 287bd1cfeb1e4e709d40fb3368aaca77 (table rebalance_test_table_4 [id=886d332d90d54101ad91fcaa63185b7a]) successfully deleted
I20240503 01:59:20.459372 3678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40506
I20240503 01:59:20.459517 3678 raft_consensus.cc:595] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.469671 3938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52430
I20240503 01:59:20.469818 3938 raft_consensus.cc:595] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.490772 3678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40526
I20240503 01:59:20.490918 3678 raft_consensus.cc:595] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.501384 3938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52460
I20240503 01:59:20.501519 3938 raft_consensus.cc:595] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.712798 3678 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6"
dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40542
I20240503 01:59:20.712952 3678 raft_consensus.cc:595] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.723608 3938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129"
dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52486
I20240503 01:59:20.723776 3938 raft_consensus.cc:595] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:20.744220 6773 raft_consensus.cc:979] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: : Instructing follower f929df19fdfc4df1986e990c60056733 to start an election
I20240503 01:59:20.744331 6773 raft_consensus.cc:1067] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Signalling peer f929df19fdfc4df1986e990c60056733 to start an election
I20240503 01:59:20.744798 3809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129"
dest_uuid: "f929df19fdfc4df1986e990c60056733"
from {username='slave'} at 127.0.99.131:36823
I20240503 01:59:20.744962 3809 raft_consensus.cc:482] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:20.745030 3809 raft_consensus.cc:3037] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.745863 3809 raft_consensus.cc:504] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.746299 3809 leader_election.cc:288] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 2 election: Requested vote from peers c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705), 11863510d4f04be8becdb64e1dff19cc (127.0.99.136:39581)
I20240503 01:59:20.746696 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 2 candidate_status { last_received { term: 1 index: 199 } } ignore_live_leader: true dest_uuid: "c8595f9ae8f644d6901791810d7b970e"
I20240503 01:59:20.746824 3938 raft_consensus.cc:3032] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:20.746872 3938 raft_consensus.cc:729] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8595f9ae8f644d6901791810d7b970e, State: Running, Role: LEADER
I20240503 01:59:20.747015 3938 consensus_queue.cc:260] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 1.199, Last appended by leader: 199, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.747207 3938 raft_consensus.cc:3037] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.748099 3938 raft_consensus.cc:2454] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 2.
I20240503 01:59:20.748446 3743 leader_election.cc:302] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8595f9ae8f644d6901791810d7b970e, f929df19fdfc4df1986e990c60056733; no voters:
I20240503 01:59:20.748628 6300 raft_consensus.cc:2781] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:20.748768 6300 raft_consensus.cc:686] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 2 LEADER]: Becoming Leader. State: Replica: f929df19fdfc4df1986e990c60056733, State: Running, Role: LEADER
I20240503 01:59:20.748926 6300 consensus_queue.cc:237] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 1.199, Last appended by leader: 199, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.750262 6421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aa056182cf435d8a6f675aa2b42129" candidate_uuid: "f929df19fdfc4df1986e990c60056733" candidate_term: 2 candidate_status { last_received { term: 1 index: 199 } } ignore_live_leader: true dest_uuid: "11863510d4f04be8becdb64e1dff19cc"
I20240503 01:59:20.750447 6421 raft_consensus.cc:3037] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.750564 3529 catalog_manager.cc:5484] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 reported cstate change: term changed from 1 to 2, leader changed from c8595f9ae8f644d6901791810d7b970e (127.0.99.131) to f929df19fdfc4df1986e990c60056733 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:20.751456 6421 raft_consensus.cc:2454] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f929df19fdfc4df1986e990c60056733 in term 2.
I20240503 01:59:20.756585 6726 raft_consensus.cc:979] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: : Instructing follower 11863510d4f04be8becdb64e1dff19cc to start an election
I20240503 01:59:20.756695 6727 raft_consensus.cc:1067] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Signalling peer 11863510d4f04be8becdb64e1dff19cc to start an election
I20240503 01:59:20.757313 6421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6"
dest_uuid: "11863510d4f04be8becdb64e1dff19cc"
from {username='slave'} at 127.0.99.129:56093
I20240503 01:59:20.757481 6421 raft_consensus.cc:482] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:20.757544 6421 raft_consensus.cc:3037] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.758347 6421 raft_consensus.cc:504] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.759302 6421 leader_election.cc:288] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [CANDIDATE]: Term 2 election: Requested vote from peers 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461), 7a367744629a43edb0df92463a498258 (127.0.99.134:42343)
I20240503 01:59:20.759620 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6" candidate_uuid: "11863510d4f04be8becdb64e1dff19cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 256 } } ignore_live_leader: true dest_uuid: "2820cbd55ee1470880e35a8ab671563b"
I20240503 01:59:20.759747 3678 raft_consensus.cc:3032] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:20.759795 3678 raft_consensus.cc:729] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2820cbd55ee1470880e35a8ab671563b, State: Running, Role: LEADER
I20240503 01:59:20.759727 5186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d39433e68b4bd7b14b42e850f42ae6" candidate_uuid: "11863510d4f04be8becdb64e1dff19cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 256 } } ignore_live_leader: true dest_uuid: "7a367744629a43edb0df92463a498258"
I20240503 01:59:20.759920 5186 raft_consensus.cc:3037] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.759950 3678 consensus_queue.cc:260] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 256, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.760113 3678 raft_consensus.cc:3037] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:20.760733 5186 raft_consensus.cc:2454] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11863510d4f04be8becdb64e1dff19cc in term 2.
I20240503 01:59:20.760880 3678 raft_consensus.cc:2454] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11863510d4f04be8becdb64e1dff19cc in term 2.
I20240503 01:59:20.761143 6356 leader_election.cc:302] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11863510d4f04be8becdb64e1dff19cc, 7a367744629a43edb0df92463a498258; no voters:
I20240503 01:59:20.761477 6681 raft_consensus.cc:2781] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:20.761636 6681 raft_consensus.cc:686] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEADER]: Becoming Leader. State: Replica: 11863510d4f04be8becdb64e1dff19cc, State: Running, Role: LEADER
I20240503 01:59:20.761901 6681 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 256, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:20.763832 3529 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc reported cstate change: term changed from 1 to 2, leader changed from 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) to 11863510d4f04be8becdb64e1dff19cc (127.0.99.136). New cstate: current_term: 2 leader_uuid: "11863510d4f04be8becdb64e1dff19cc" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:21.262171 3938 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 2 index: 200. (index mismatch)
I20240503 01:59:21.262692 6300 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8595f9ae8f644d6901791810d7b970e" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
I20240503 01:59:21.267227 6421 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 2 index: 200. (index mismatch)
I20240503 01:59:21.267665 6890 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
I20240503 01:59:21.322886 3678 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Refusing update from remote peer 11863510d4f04be8becdb64e1dff19cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20240503 01:59:21.323522 6893 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2820cbd55ee1470880e35a8ab671563b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 01:59:21.326458 5186 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Refusing update from remote peer 11863510d4f04be8becdb64e1dff19cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20240503 01:59:21.326884 6895 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 01:59:21.755322 6421 consensus_queue.cc:237] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 256, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:21.756469 5186 raft_consensus.cc:1261] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Refusing update from remote peer 11863510d4f04be8becdb64e1dff19cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20240503 01:59:21.756896 6897 consensus_queue.cc:1035] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 01:59:21.758553 6893 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 256 to 258, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:21.759050 5186 raft_consensus.cc:2932] T f8d39433e68b4bd7b14b42e850f42ae6 P 7a367744629a43edb0df92463a498258 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 256 to 258, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:21.760238 3523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8d39433e68b4bd7b14b42e850f42ae6 with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:21.761219 3529 catalog_manager.cc:5484] T f8d39433e68b4bd7b14b42e850f42ae6 P 11863510d4f04be8becdb64e1dff19cc reported cstate change: config changed from index 256 to 258, VOTER 2820cbd55ee1470880e35a8ab671563b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "11863510d4f04be8becdb64e1dff19cc" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "7a367744629a43edb0df92463a498258" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:21.764006 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d39433e68b4bd7b14b42e850f42ae6 with delete_type TABLET_DATA_TOMBSTONED (TS 2820cbd55ee1470880e35a8ab671563b not found in new config with opid_index 258) from {username='slave'} at 127.0.0.1:60546
I20240503 01:59:21.764394 6924 tablet_replica.cc:330] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: stopping tablet replica
I20240503 01:59:21.764555 6924 raft_consensus.cc:2229] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:21.764676 6924 raft_consensus.cc:2258] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:21.766258 6924 ts_tablet_manager.cc:1875] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:21.768414 6924 ts_tablet_manager.cc:1888] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.257
I20240503 01:59:21.768483 6924 log.cc:1196] T f8d39433e68b4bd7b14b42e850f42ae6 P 2820cbd55ee1470880e35a8ab671563b: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:21.769117 3521 catalog_manager.cc:4830] TS 2820cbd55ee1470880e35a8ab671563b (127.0.99.129:43461): tablet f8d39433e68b4bd7b14b42e850f42ae6 (table rebalance_test_table_1 [id=d6f2db11fd294444b563e9f8ee202eaf]) successfully deleted
I20240503 01:59:21.864048 3809 consensus_queue.cc:237] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 2.200, Last appended by leader: 199, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } }
I20240503 01:59:21.865032 6421 raft_consensus.cc:1261] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Refusing update from remote peer f929df19fdfc4df1986e990c60056733: Log matching property violated. Preceding OpId in replica: term: 2 index: 200. Preceding OpId from leader: term: 2 index: 201. (index mismatch)
I20240503 01:59:21.865432 6943 consensus_queue.cc:1035] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240503 01:59:21.867034 6890 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 [term 2 LEADER]: Committing config change with OpId 2.201: config changed from index 199 to 201, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:21.867435 6421 raft_consensus.cc:2932] T b1aa056182cf435d8a6f675aa2b42129 P 11863510d4f04be8becdb64e1dff19cc [term 2 FOLLOWER]: Committing config change with OpId 2.201: config changed from index 199 to 201, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } } }
I20240503 01:59:21.868767 3520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1aa056182cf435d8a6f675aa2b42129 with cas_config_opid_index 199: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:21.869541 3532 catalog_manager.cc:5484] T b1aa056182cf435d8a6f675aa2b42129 P f929df19fdfc4df1986e990c60056733 reported cstate change: config changed from index 199 to 201, VOTER c8595f9ae8f644d6901791810d7b970e (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "f929df19fdfc4df1986e990c60056733" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "f929df19fdfc4df1986e990c60056733" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11863510d4f04be8becdb64e1dff19cc" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39581 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:21.871822 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet b1aa056182cf435d8a6f675aa2b42129 with delete_type TABLET_DATA_TOMBSTONED (TS c8595f9ae8f644d6901791810d7b970e not found in new config with opid_index 201) from {username='slave'} at 127.0.0.1:46500
I20240503 01:59:21.872205 6944 tablet_replica.cc:330] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: stopping tablet replica
I20240503 01:59:21.872339 6944 raft_consensus.cc:2229] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:21.872449 6944 raft_consensus.cc:2258] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:21.873800 6944 ts_tablet_manager.cc:1875] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:21.875696 6944 ts_tablet_manager.cc:1888] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.200
I20240503 01:59:21.875795 6944 log.cc:1196] T b1aa056182cf435d8a6f675aa2b42129 P c8595f9ae8f644d6901791810d7b970e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:21.876299 3521 catalog_manager.cc:4830] TS c8595f9ae8f644d6901791810d7b970e (127.0.99.131:36705): tablet b1aa056182cf435d8a6f675aa2b42129 (table rebalance_test_table_2 [id=0b4675bc7a7349dd9e5688f19dce2219]) successfully deleted
I20240503 01:59:22.060249 6401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.060506 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.066381 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.071372 3657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.078372 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.089361 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.095392 3920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.098019 3788 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.179490 6401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.187245 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.188122 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.190853 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.191613 3657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.191879 3920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.205194 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.211644 3788 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.309793 4900 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.310227 5298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.311606 5167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:22.312433 6401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.313248 5024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:22.315734 3788 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.320691 3657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:22.339165 3920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d88debf910664f928bbd978e084af0e0 | 127.0.99.190:33221 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39243 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0fb6a1bb16214e3c9ec6acd1ab111358 | 127.0.99.135:42189 | HEALTHY | <none> | 1 | 0
11863510d4f04be8becdb64e1dff19cc | 127.0.99.136:39581 | HEALTHY | <none> | 1 | 0
2820cbd55ee1470880e35a8ab671563b | 127.0.99.129:43461 | HEALTHY | <none> | 3 | 0
7a367744629a43edb0df92463a498258 | 127.0.99.134:42343 | HEALTHY | <none> | 2 | 0
7b5c836ac65b442e808f46fb2d267d2c | 127.0.99.132:35901 | HEALTHY | <none> | 2 | 0
c6f646ec3edd4e298f16122028bb0ef7 | 127.0.99.133:41793 | HEALTHY | <none> | 1 | 0
c8595f9ae8f644d6901791810d7b970e | 127.0.99.131:36705 | HEALTHY | <none> | 3 | 0
f929df19fdfc4df1986e990c60056733 | 127.0.99.130:35911 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43461
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:35911
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:36705
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35901
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41793
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:42343
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:42189
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:39581
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43461
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:35911
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:36705
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35901
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41793
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:42343
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:42189
server_dump_info_path | /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:39581
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39243 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:22.371405 398 log_verifier.cc:126] Checking tablet 02a421366d744afd99d7779644e9e41e
I20240503 01:59:22.415266 398 log_verifier.cc:177] Verified matching terms for 222 ops in tablet 02a421366d744afd99d7779644e9e41e
I20240503 01:59:22.415419 398 log_verifier.cc:126] Checking tablet 1c6b5cb588234440807eee9875c334c1
I20240503 01:59:22.469883 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet 1c6b5cb588234440807eee9875c334c1
I20240503 01:59:22.470023 398 log_verifier.cc:126] Checking tablet 287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:22.514700 398 log_verifier.cc:177] Verified matching terms for 230 ops in tablet 287bd1cfeb1e4e709d40fb3368aaca77
I20240503 01:59:22.514837 398 log_verifier.cc:126] Checking tablet 4baa1b72653c4a0ba130008b06bc5897
I20240503 01:59:22.556421 398 log_verifier.cc:177] Verified matching terms for 232 ops in tablet 4baa1b72653c4a0ba130008b06bc5897
I20240503 01:59:22.556552 398 log_verifier.cc:126] Checking tablet 4d2ab9fbe93a44adbc34f306e8d61c27
I20240503 01:59:22.596873 398 log_verifier.cc:177] Verified matching terms for 203 ops in tablet 4d2ab9fbe93a44adbc34f306e8d61c27
I20240503 01:59:22.596990 398 log_verifier.cc:126] Checking tablet 56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:22.646350 398 log_verifier.cc:177] Verified matching terms for 237 ops in tablet 56cd6bd86d2a4391947f7b7193b5e420
I20240503 01:59:22.646486 398 log_verifier.cc:126] Checking tablet 5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:22.687515 398 log_verifier.cc:177] Verified matching terms for 217 ops in tablet 5dba64ef1acb4293be37a986fc18a105
I20240503 01:59:22.687675 398 log_verifier.cc:126] Checking tablet 6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:22.729070 398 log_verifier.cc:177] Verified matching terms for 228 ops in tablet 6365fecf9fa04d6eaf6ce9dbe47add84
I20240503 01:59:22.729182 398 log_verifier.cc:126] Checking tablet 6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:22.775298 398 log_verifier.cc:177] Verified matching terms for 254 ops in tablet 6a7cf094bb2546bb800180d3a530f46a
I20240503 01:59:22.775417 398 log_verifier.cc:126] Checking tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:22.819953 398 log_verifier.cc:177] Verified matching terms for 229 ops in tablet a67fef22e6eb4ecebb2ba7c4d7a5b0c5
I20240503 01:59:22.820068 398 log_verifier.cc:126] Checking tablet b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:22.861440 398 log_verifier.cc:177] Verified matching terms for 201 ops in tablet b1aa056182cf435d8a6f675aa2b42129
I20240503 01:59:22.861553 398 log_verifier.cc:126] Checking tablet cbba240087984f5bac2370d46a09a436
I20240503 01:59:22.908138 398 log_verifier.cc:177] Verified matching terms for 232 ops in tablet cbba240087984f5bac2370d46a09a436
I20240503 01:59:22.908249 398 log_verifier.cc:126] Checking tablet e504950788d7408a83d4b7299b78518c
I20240503 01:59:22.948803 398 log_verifier.cc:177] Verified matching terms for 216 ops in tablet e504950788d7408a83d4b7299b78518c
I20240503 01:59:22.948920 398 log_verifier.cc:126] Checking tablet e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:22.989750 398 log_verifier.cc:177] Verified matching terms for 214 ops in tablet e6c983adafca4912aa70f99c81bb4b4f
I20240503 01:59:22.989852 398 log_verifier.cc:126] Checking tablet ea7e73129def488bbf9ed086e424968e
I20240503 01:59:22.998955 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ea7e73129def488bbf9ed086e424968e
I20240503 01:59:22.999015 398 log_verifier.cc:126] Checking tablet f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:23.047186 398 log_verifier.cc:177] Verified matching terms for 258 ops in tablet f8d39433e68b4bd7b14b42e850f42ae6
I20240503 01:59:23.048782 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3596
I20240503 01:59:23.057242 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3727
I20240503 01:59:23.063874 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3858
I20240503 01:59:23.070760 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 4837
I20240503 01:59:23.079145 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 4969
I20240503 01:59:23.086318 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 5105
I20240503 01:59:23.091713 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 5236
I20240503 01:59:23.096946 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 6339
I20240503 01:59:23.102805 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 3505
2024-05-03T01:59:23Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (21989 ms)
[----------] 1 test from RebalanceParamTest (21989 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 01:59:23.158319 398 test_util.cc:268] Using random seed: 294739753
I20240503 01:59:23.159138 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:23.159178 398 ts_itest-base.cc:113] --------------
I20240503 01:59:23.159188 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 01:59:23.159198 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:23.159209 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:23Z Disabled control of system clock
I20240503 01:59:23.167275 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41207
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:41207
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:59:23.275523 7068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:23.275813 7068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:23.275854 7068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:23.278852 7068 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:23.278928 7068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:23.278952 7068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:23.278970 7068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:23.278986 7068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:23.282687 7068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:41207
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:23.283612 7068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:23.284734 7068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:23.292774 7074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.293846 7076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.294795 7073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:23.295780 7068 server_base.cc:1034] running on GCE node
I20240503 01:59:23.296254 7068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:23.297176 7068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:23.298329 7068 hybrid_clock.cc:648] HybridClock initialized: now 1714701563298317 us; error 24 us; skew 500 ppm
I20240503 01:59:23.305963 7068 webserver.cc:460] Webserver started at http://127.0.99.190:34541/ using document root <none> and password file <none>
I20240503 01:59:23.306661 7068 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:23.306726 7068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:23.306914 7068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:23.308655 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b417ed481cd9481ba8418c1658ec8f19"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.309123 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b417ed481cd9481ba8418c1658ec8f19"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.312574 7068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 01:59:23.314817 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.316130 7068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:23.316265 7068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b417ed481cd9481ba8418c1658ec8f19"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.316355 7068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:23.329267 7068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:23.329895 7068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:23.330044 7068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:23.336632 7068 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:41207
I20240503 01:59:23.336637 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:41207 every 8 connection(s)
I20240503 01:59:23.337579 7068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:23.339982 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.343725 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7068
I20240503 01:59:23.343869 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:23.345263 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19: Bootstrap starting.
I20240503 01:59:23.347364 7135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.348160 7135 log.cc:824] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.349776 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19: No bootstrap required, opened a new log
I20240503 01:59:23.352341 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } }
I20240503 01:59:23.352519 7135 raft_consensus.cc:374] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.352569 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b417ed481cd9481ba8418c1658ec8f19, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.353013 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } }
I20240503 01:59:23.353137 7135 raft_consensus.cc:388] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:23.353179 7135 raft_consensus.cc:482] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:23.353253 7135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.353935 7135 raft_consensus.cc:504] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } }
I20240503 01:59:23.354219 7135 leader_election.cc:302] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b417ed481cd9481ba8418c1658ec8f19; no voters:
I20240503 01:59:23.354463 7135 leader_election.cc:288] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:23.354569 7140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.354770 7140 raft_consensus.cc:686] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [term 1 LEADER]: Becoming Leader. State: Replica: b417ed481cd9481ba8418c1658ec8f19, State: Running, Role: LEADER
I20240503 01:59:23.355083 7140 consensus_queue.cc:237] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } }
I20240503 01:59:23.355494 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:23.356907 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b417ed481cd9481ba8418c1658ec8f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } } }
I20240503 01:59:23.357013 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:23.357231 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b417ed481cd9481ba8418c1658ec8f19. Latest consensus state: current_term: 1 leader_uuid: "b417ed481cd9481ba8418c1658ec8f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b417ed481cd9481ba8418c1658ec8f19" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41207 } } }
I20240503 01:59:23.357343 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:23.357362 7150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:23.360203 7150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:23.365260 7150 catalog_manager.cc:1347] Generated new cluster ID: 8d020e2d581a4b47bf7c27c6a166e30f
I20240503 01:59:23.365342 7150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:23.379271 7150 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:23.380513 7150 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:23.388660 7150 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b417ed481cd9481ba8418c1658ec8f19: Generated new TSK 0
I20240503 01:59:23.389380 7150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:23.398372 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:23.521984 7159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:23.522238 7159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:23.522312 7159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:23.525414 7159 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:23.525509 7159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:23.525599 7159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:23.529323 7159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:23.530252 7159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:23.531348 7159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:23.540619 7164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.541368 7167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.542456 7165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:23.543452 7159 server_base.cc:1034] running on GCE node
I20240503 01:59:23.543923 7159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:23.544524 7159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:23.545692 7159 hybrid_clock.cc:648] HybridClock initialized: now 1714701563545677 us; error 32 us; skew 500 ppm
I20240503 01:59:23.547595 7159 webserver.cc:460] Webserver started at http://127.0.99.129:46605/ using document root <none> and password file <none>
I20240503 01:59:23.548142 7159 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:23.548202 7159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:23.548372 7159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:23.549970 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.550416 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.553678 7159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:23.555847 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.556877 7159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:23.556978 7159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.557067 7159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:23.573714 7159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:23.574355 7159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:23.574488 7159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:23.574950 7159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:23.575822 7159 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:23.575878 7159 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.575927 7159 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:23.575947 7159 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.584615 7159 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:42469
I20240503 01:59:23.584663 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:42469 every 8 connection(s)
I20240503 01:59:23.585561 7159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:23.594388 7287 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:23.594647 7287 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:23.594827 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7159
I20240503 01:59:23.594946 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:23.595230 7287 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:23.596668 7098 ts_manager.cc:194] Registered new tserver with Master: 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:23.597023 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:23.597837 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48719
W20240503 01:59:23.705190 7290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:23.705421 7290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:23.705493 7290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:23.708477 7290 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:23.708552 7290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:23.708653 7290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:23.712217 7290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:23.713202 7290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:23.714385 7290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:23.723419 7298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.724164 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.725334 7296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:23.726280 7290 server_base.cc:1034] running on GCE node
I20240503 01:59:23.726703 7290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:23.727280 7290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:23.728430 7290 hybrid_clock.cc:648] HybridClock initialized: now 1714701563728373 us; error 75 us; skew 500 ppm
I20240503 01:59:23.730451 7290 webserver.cc:460] Webserver started at http://127.0.99.130:35449/ using document root <none> and password file <none>
I20240503 01:59:23.731037 7290 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:23.731093 7290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:23.731277 7290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:23.733031 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bed387c0ff9241688cdf45f364dc6f5a"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.733505 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bed387c0ff9241688cdf45f364dc6f5a"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.736826 7290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:23.739128 7304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.740255 7290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:23.740375 7290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bed387c0ff9241688cdf45f364dc6f5a"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.740463 7290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:23.757969 7290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:23.758633 7290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:23.758805 7290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:23.759352 7290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:23.760360 7290 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:23.760411 7290 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.760473 7290 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:23.760501 7290 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.769275 7290 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46181
I20240503 01:59:23.769304 7417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46181 every 8 connection(s)
I20240503 01:59:23.770255 7290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:23.773633 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7290
I20240503 01:59:23.773780 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:23.776214 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:23.780068 7418 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:23.780325 7418 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:23.781028 7418 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:23.782133 7098 ts_manager.cc:194] Registered new tserver with Master: bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:23.782677 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:44213
W20240503 01:59:23.885802 7421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:23.886040 7421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:23.886086 7421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:23.889281 7421 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:23.889358 7421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:23.889459 7421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:23.893255 7421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:23.894217 7421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:23.895344 7421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:23.904398 7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.905380 7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:23.906260 7429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:23.907389 7421 server_base.cc:1034] running on GCE node
I20240503 01:59:23.907858 7421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:23.908430 7421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:23.909584 7421 hybrid_clock.cc:648] HybridClock initialized: now 1714701563909572 us; error 27 us; skew 500 ppm
I20240503 01:59:23.911561 7421 webserver.cc:460] Webserver started at http://127.0.99.131:35033/ using document root <none> and password file <none>
I20240503 01:59:23.912104 7421 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:23.912156 7421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:23.912313 7421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:23.913873 7421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.914286 7421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.917443 7421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:23.919466 7435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.920523 7421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:23.920624 7421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
format_stamp: "Formatted at 2024-05-03 01:59:23 on dist-test-slave-pvfz"
I20240503 01:59:23.920693 7421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:23.935106 7421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:23.935690 7421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:23.935822 7421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:23.936271 7421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:23.937074 7421 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:23.937119 7421 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.937155 7421 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:23.937175 7421 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:23.945269 7421 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34921
I20240503 01:59:23.945315 7548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34921 every 8 connection(s)
I20240503 01:59:23.946202 7421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:23.953323 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7421
I20240503 01:59:23.953467 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:23.955444 7549 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:23.955734 7549 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:23.956414 7549 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:23.957031 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:23.957584 7098 ts_manager.cc:194] Registered new tserver with Master: 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:23.958258 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:37135
W20240503 01:59:24.064246 7552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.064473 7552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.064517 7552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.067512 7552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.067631 7552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.067727 7552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:24.071273 7552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.072222 7552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.073331 7552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.082626 7560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.083755 7558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.084615 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.085748 7552 server_base.cc:1034] running on GCE node
I20240503 01:59:24.086158 7552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.086748 7552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.087919 7552 hybrid_clock.cc:648] HybridClock initialized: now 1714701564087891 us; error 41 us; skew 500 ppm
I20240503 01:59:24.090149 7552 webserver.cc:460] Webserver started at http://127.0.99.132:45273/ using document root <none> and password file <none>
I20240503 01:59:24.090796 7552 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.090866 7552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.091058 7552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.092880 7552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "efeb87c28ee04b3da5c686665b0c5c58"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.093351 7552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "efeb87c28ee04b3da5c686665b0c5c58"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.096755 7552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:24.098948 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.100008 7552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.100121 7552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "efeb87c28ee04b3da5c686665b0c5c58"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.100204 7552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.114310 7552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.115032 7552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.115231 7552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.115932 7552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.116909 7552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.116967 7552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.117031 7552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.117071 7552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.126368 7552 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:33123
I20240503 01:59:24.126410 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:33123 every 8 connection(s)
I20240503 01:59:24.127307 7552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:24.132992 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7552
I20240503 01:59:24.133119 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:24.135506 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.136612 7680 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:24.136878 7680 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.137645 7680 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:24.138677 7098 ts_manager.cc:194] Registered new tserver with Master: efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:24.139261 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:58951
W20240503 01:59:24.245373 7683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.245627 7683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.245695 7683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.248849 7683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.248934 7683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.249033 7683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:24.252831 7683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.253798 7683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.254930 7683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.264083 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.264752 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.265870 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.266764 7683 server_base.cc:1034] running on GCE node
I20240503 01:59:24.267172 7683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.267782 7683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.268923 7683 hybrid_clock.cc:648] HybridClock initialized: now 1714701564268902 us; error 39 us; skew 500 ppm
I20240503 01:59:24.270851 7683 webserver.cc:460] Webserver started at http://127.0.99.133:34155/ using document root <none> and password file <none>
I20240503 01:59:24.271370 7683 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.271421 7683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.271617 7683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.273444 7683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.273919 7683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.277227 7683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 01:59:24.279516 7697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.280560 7683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:24.280649 7683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.280723 7683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.304471 7683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.305076 7683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.305209 7683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.305680 7683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.306484 7683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.306529 7683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.306576 7683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.306591 7683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.314680 7683 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34145
I20240503 01:59:24.314711 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34145 every 8 connection(s)
I20240503 01:59:24.315685 7683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:24.321998 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7683
I20240503 01:59:24.322137 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:24.324708 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.325075 7811 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:24.325306 7811 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.325856 7811 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:24.326917 7098 ts_manager.cc:194] Registered new tserver with Master: e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145)
I20240503 01:59:24.327445 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57119
W20240503 01:59:24.435880 7814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.436121 7814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.436169 7814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.439312 7814 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.439404 7814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.439482 7814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:24.443253 7814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.444347 7814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.445545 7814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.454612 7822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.455652 7820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.456447 7819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.457397 7814 server_base.cc:1034] running on GCE node
I20240503 01:59:24.457923 7814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.458484 7814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.459609 7814 hybrid_clock.cc:648] HybridClock initialized: now 1714701564459589 us; error 38 us; skew 500 ppm
I20240503 01:59:24.461678 7814 webserver.cc:460] Webserver started at http://127.0.99.134:38471/ using document root <none> and password file <none>
I20240503 01:59:24.462265 7814 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.462322 7814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.462493 7814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.464350 7814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.464869 7814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.468389 7814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:24.470736 7828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.471843 7814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.471951 7814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.472043 7814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.481781 7814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.482350 7814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.482517 7814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.483042 7814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.484027 7814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.484077 7814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.484143 7814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.484170 7814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.492996 7814 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43615
I20240503 01:59:24.493036 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43615 every 8 connection(s)
I20240503 01:59:24.493959 7814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:24.500931 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7814
I20240503 01:59:24.501051 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:24.503067 7942 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:24.503294 7942 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.503463 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.503926 7942 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:24.504925 7099 ts_manager.cc:194] Registered new tserver with Master: c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615)
I20240503 01:59:24.505476 7099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:53673
I20240503 01:59:24.600122 7287 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
W20240503 01:59:24.615531 7945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.615844 7945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.615895 7945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.618932 7945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.619009 7945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.619079 7945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:24.622826 7945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.7945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.623801 7945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.625007 7945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.634405 7953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.635452 7951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.636344 7950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.637370 7945 server_base.cc:1034] running on GCE node
I20240503 01:59:24.637740 7945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.638295 7945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.639456 7945 hybrid_clock.cc:648] HybridClock initialized: now 1714701564639438 us; error 32 us; skew 500 ppm
I20240503 01:59:24.641674 7945 webserver.cc:460] Webserver started at http://127.0.99.135:45419/ using document root <none> and password file <none>
I20240503 01:59:24.642199 7945 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.642251 7945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.642398 7945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.644090 7945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c778431b13784c7c81f3924b7f73e664"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.644518 7945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c778431b13784c7c81f3924b7f73e664"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.648216 7945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:24.650501 7959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.651698 7945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.651811 7945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c778431b13784c7c81f3924b7f73e664"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.651901 7945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.680738 7945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.681345 7945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.681479 7945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.681993 7945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.682832 7945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.682875 7945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.682914 7945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.682929 7945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.691591 7945 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43981
I20240503 01:59:24.691633 8072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43981 every 8 connection(s)
I20240503 01:59:24.692555 7945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:24.699841 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 7945
I20240503 01:59:24.699968 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:24.700912 8073 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:24.701117 8073 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.701711 8073 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:24.702157 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:24.702700 7099 ts_manager.cc:194] Registered new tserver with Master: c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:24.703263 7099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:46403
I20240503 01:59:24.785145 7418 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
W20240503 01:59:24.810264 8076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.810490 8076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:24.810540 8076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.813609 8076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:24.813681 8076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:24.813753 8076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:24.817402 8076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.8076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:24.818320 8076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:24.819464 8076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:24.828305 8084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.829397 8082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:24.830440 8081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:24.831378 8076 server_base.cc:1034] running on GCE node
I20240503 01:59:24.831820 8076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:24.832343 8076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:24.833498 8076 hybrid_clock.cc:648] HybridClock initialized: now 1714701564833490 us; error 35 us; skew 500 ppm
I20240503 01:59:24.835348 8076 webserver.cc:460] Webserver started at http://127.0.99.136:41281/ using document root <none> and password file <none>
I20240503 01:59:24.835933 8076 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:24.835989 8076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:24.836138 8076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:24.837832 8076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "72577fce0313400ea862bf873738af54"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.838265 8076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "72577fce0313400ea862bf873738af54"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.841706 8076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:24.844151 8090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.845366 8076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:24.845479 8076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "72577fce0313400ea862bf873738af54"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:24.845602 8076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:24.857406 8076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:24.858065 8076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:24.858218 8076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:24.858809 8076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:24.859763 8076 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:24.859810 8076 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.859862 8076 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:24.859894 8076 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:24.868180 8076 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:33955
I20240503 01:59:24.868206 8203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:33955 every 8 connection(s)
I20240503 01:59:24.869136 8076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:24.878233 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 8076
I20240503 01:59:24.878273 8204 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:24.878381 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:24.878552 8204 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:24.879184 8204 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:24.880192 7098 ts_manager.cc:194] Registered new tserver with Master: 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:24.880781 7098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34603
I20240503 01:59:24.892791 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:24.902635 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:24.902720 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 7683
I20240503 01:59:24.908299 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 7814
I20240503 01:59:24.913403 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 7945
I20240503 01:59:24.918699 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8Tfalf/build/debug/bin/kudu with pid 8076
I20240503 01:59:24.960330 7549 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:25.141502 7680 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:28.675431 7098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:28.696484 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 68791b3adccd4168b55a2da4423232d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.696497 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 9796308eacd94fb5ad3cb524aa590cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.696497 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 68791b3adccd4168b55a2da4423232d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.697414 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.697419 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.697419 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9796308eacd94fb5ad3cb524aa590cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.699877 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 18468345c7924d4f88366524b56e8ab3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.700038 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 9796308eacd94fb5ad3cb524aa590cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.700139 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18468345c7924d4f88366524b56e8ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.700295 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9796308eacd94fb5ad3cb524aa590cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.700335 7217 tablet_service.cc:1467] Processing CreateTablet for tablet 18468345c7924d4f88366524b56e8ab3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.700574 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18468345c7924d4f88366524b56e8ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.702456 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 9796308eacd94fb5ad3cb524aa590cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.703424 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9796308eacd94fb5ad3cb524aa590cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.703763 7351 tablet_service.cc:1467] Processing CreateTablet for tablet 18468345c7924d4f88366524b56e8ab3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.704172 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 68791b3adccd4168b55a2da4423232d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.706275 8226 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:28.706696 7351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18468345c7924d4f88366524b56e8ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.706887 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.709924 8224 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:28.710101 8226 tablet_bootstrap.cc:654] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.710963 8226 log.cc:824] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:28.711403 8227 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:28.711697 8225 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:28.712021 8224 tablet_bootstrap.cc:654] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.712805 8224 log.cc:824] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:28.713121 8226 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:28.713356 8226 ts_tablet_manager.cc:1367] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 01:59:28.713479 8227 tablet_bootstrap.cc:654] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.714538 8225 tablet_bootstrap.cc:654] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.714644 8224 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:28.714596 8227 log.cc:824] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:28.714846 8224 ts_tablet_manager.cc:1367] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:28.715323 8225 log.cc:824] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:28.716348 8227 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:28.716500 8227 ts_tablet_manager.cc:1367] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:28.716864 8225 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:28.717036 8225 ts_tablet_manager.cc:1367] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:28.717407 8226 raft_consensus.cc:348] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.717603 8226 raft_consensus.cc:374] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.717648 8226 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.718107 8226 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.718953 8226 ts_tablet_manager.cc:1398] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:59:28.719367 8227 raft_consensus.cc:348] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.719599 8224 raft_consensus.cc:348] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.719846 8227 raft_consensus.cc:374] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.719851 8224 raft_consensus.cc:374] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.719908 8224 raft_consensus.cc:729] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.719954 8227 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.720432 8224 consensus_queue.cc:260] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.720443 8227 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.720696 8225 raft_consensus.cc:348] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.720887 8225 raft_consensus.cc:374] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.720947 8225 raft_consensus.cc:729] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.721293 8227 ts_tablet_manager.cc:1398] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240503 01:59:28.721315 8225 consensus_queue.cc:260] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.721498 8224 ts_tablet_manager.cc:1398] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 01:59:28.721911 8224 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:28.722054 8225 ts_tablet_manager.cc:1398] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240503 01:59:28.722355 8225 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:28.723088 8224 tablet_bootstrap.cc:654] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.723320 8225 tablet_bootstrap.cc:654] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.724053 8227 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:28.724850 8226 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:28.725109 8227 tablet_bootstrap.cc:654] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.726320 8226 tablet_bootstrap.cc:654] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.726329 8225 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:28.726409 8225 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:28.727034 8227 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:28.726969 8225 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.727093 8225 raft_consensus.cc:374] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.727108 8227 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:28.727128 8225 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.727270 8225 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.727602 8225 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.727708 8227 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.727823 8227 raft_consensus.cc:374] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.727869 8227 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.727979 8227 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.728536 8227 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.728567 8224 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:28.728647 8224 ts_tablet_manager.cc:1367] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 01:59:28.728832 8227 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:28.729202 8224 raft_consensus.cc:348] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.729302 8224 raft_consensus.cc:374] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.729337 8224 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.729445 8226 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:28.729521 8226 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 01:59:28.729486 8224 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.729764 8227 tablet_bootstrap.cc:654] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.729764 8224 ts_tablet_manager.cc:1398] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:28.730019 8226 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.730129 8226 raft_consensus.cc:374] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.730177 8226 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.730288 8226 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:28.730556 8226 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:28.731473 8227 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:28.731608 8227 ts_tablet_manager.cc:1367] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:28.732295 8227 raft_consensus.cc:348] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.732414 8227 raft_consensus.cc:374] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.732498 8227 raft_consensus.cc:729] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.732650 8227 consensus_queue.cc:260] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.732985 8227 ts_tablet_manager.cc:1398] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.755842 8233 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.756033 8233 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.757249 8233 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:28.762214 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:28.762214 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:28.762511 7634 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:28.762511 7241 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:28.762951 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:28.763240 8233 raft_consensus.cc:2781] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:28.763314 8233 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:28.763368 8233 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.764212 8233 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.764518 8233 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:28.764832 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:28.764832 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:28.764946 7634 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.764946 7241 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.765990 7241 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:28.765998 7634 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:28.766266 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:28.766410 8233 raft_consensus.cc:2781] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:28.766610 8233 raft_consensus.cc:686] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:28.766880 8233 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.769886 7099 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:28.774716 7419 tablet.cc:2369] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:28.840090 7288 tablet.cc:2369] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:28.881958 7681 tablet.cc:2369] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:28.954334 7550 tablet.cc:2369] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:28.993340 8234 raft_consensus.cc:482] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.993480 8234 raft_consensus.cc:504] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:28.994637 8234 leader_election.cc:288] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:28.999155 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18468345c7924d4f88366524b56e8ab3" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:28.999450 7503 raft_consensus.cc:2454] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:28.999501 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18468345c7924d4f88366524b56e8ab3" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:28.999730 7372 raft_consensus.cc:2454] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:28.999850 7177 leader_election.cc:302] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:29.000078 8234 raft_consensus.cc:2781] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.000154 8234 raft_consensus.cc:482] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.000213 8234 raft_consensus.cc:3037] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.000964 8234 raft_consensus.cc:504] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:29.001276 8234 leader_election.cc:288] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:29.001565 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18468345c7924d4f88366524b56e8ab3" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:29.001621 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18468345c7924d4f88366524b56e8ab3" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:29.001658 7503 raft_consensus.cc:3037] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.001737 7372 raft_consensus.cc:3037] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.002548 7372 raft_consensus.cc:2454] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:29.002566 7503 raft_consensus.cc:2454] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:29.002813 7177 leader_election.cc:302] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:29.002976 8234 raft_consensus.cc:2781] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.003196 8234 raft_consensus.cc:686] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Running, Role: LEADER
I20240503 01:59:29.003731 8234 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:29.006727 7099 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:29.019237 8233 raft_consensus.cc:482] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.019366 8233 raft_consensus.cc:504] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.019891 8233 leader_election.cc:288] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.020385 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:29.020515 7634 raft_consensus.cc:2454] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:29.020762 7306 leader_election.cc:302] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:29.020934 8233 raft_consensus.cc:2781] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.020990 8233 raft_consensus.cc:482] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.021037 8233 raft_consensus.cc:3037] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.021797 8233 raft_consensus.cc:504] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.022122 8233 leader_election.cc:288] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.022434 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:29.022542 7634 raft_consensus.cc:3037] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.023298 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:29.023425 7634 raft_consensus.cc:2454] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:29.023416 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:29.023438 7503 raft_consensus.cc:2454] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:29.023718 7306 leader_election.cc:302] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:29.023892 8233 raft_consensus.cc:2781] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.023980 8233 raft_consensus.cc:686] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:29.024130 8233 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.026032 7099 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:29.166210 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:29.166456 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:29.169582 8250 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:29.258073 7502 raft_consensus.cc:3037] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.258184 7634 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.258178 7241 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.258195 7633 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.259478 7502 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.260449 8243 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.261252 8243 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.261539 8243 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.262144 8233 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.269237 7372 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.269707 7503 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.270298 8247 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.270709 8247 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.274077 8279 mvcc.cc:205] Tried to move back new op lower bound from 7023417627674185728 to 7023417625671462912. Current Snapshot: MvccSnapshot[applied={T|T < 7023417627674185728}]
I20240503 01:59:29.280664 8292 mvcc.cc:205] Tried to move back new op lower bound from 7023417627674132480 to 7023417626724139008. Current Snapshot: MvccSnapshot[applied={T|T < 7023417627674132480}]
I20240503 01:59:29.284541 8288 mvcc.cc:205] Tried to move back new op lower bound from 7023417627674185728 to 7023417625671462912. Current Snapshot: MvccSnapshot[applied={T|T < 7023417627674185728}]
I20240503 01:59:29.284579 8285 mvcc.cc:205] Tried to move back new op lower bound from 7023417627719139328 to 7023417626642345984. Current Snapshot: MvccSnapshot[applied={T|T < 7023417627719139328}]
INSERT report
rows total: 2000
time total: 114.969 ms
time per row: 0.0574843 ms
I20240503 01:59:29.332425 7098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:29.337527 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 81086ff737c84d9d9f04743d31c6151b (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.337859 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.337819 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 81086ff737c84d9d9f04743d31c6151b (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.338058 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.339776 8314 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:29.340286 7482 tablet_service.cc:1467] Processing CreateTablet for tablet b4628afbfeff4f8bbaba5da1e89e4850 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.340498 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.340848 8314 tablet_bootstrap.cc:654] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.340860 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 81086ff737c84d9d9f04743d31c6151b (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.341166 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.342974 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.342974 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.343226 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.343268 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.343297 7217 tablet_service.cc:1467] Processing CreateTablet for tablet b4628afbfeff4f8bbaba5da1e89e4850 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.343509 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.343856 7613 tablet_service.cc:1467] Processing CreateTablet for tablet b4628afbfeff4f8bbaba5da1e89e4850 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.344118 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.345458 7220 tablet_service.cc:1467] Processing CreateTablet for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.345685 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.346094 8317 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:29.346135 8318 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:29.347007 8318 tablet_bootstrap.cc:654] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.347019 8317 tablet_bootstrap.cc:654] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.347693 8315 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:29.348371 8317 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:29.348446 8317 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:29.348524 8315 tablet_bootstrap.cc:654] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.348874 8317 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.348966 8317 raft_consensus.cc:374] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.348994 8317 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.349139 8317 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.349418 8317 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.349550 8317 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:29.350445 8317 tablet_bootstrap.cc:654] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.350723 8315 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:29.350852 8315 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.351393 8317 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:29.351480 8317 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.351583 8318 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:29.351666 8318 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:29.351938 8317 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.352051 8317 raft_consensus.cc:374] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.352088 8317 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.351962 8315 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.352196 8315 raft_consensus.cc:374] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.352269 8315 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.352232 8317 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.352306 8318 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.352438 8318 raft_consensus.cc:374] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.352478 8318 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.352524 8317 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.352602 8318 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.352694 8317 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:29.352638 8315 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.352914 8318 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.353157 8315 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.353550 8317 tablet_bootstrap.cc:654] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.353598 8314 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:29.353689 8314 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 01:59:29.354184 8314 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.354307 8314 raft_consensus.cc:374] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.354339 8314 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.354460 8314 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.354622 8317 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:29.354696 8317 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.354746 8314 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:29.354914 8314 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:29.355150 8317 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.355260 8317 raft_consensus.cc:374] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.355302 8317 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.355404 8317 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.355697 8317 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.355949 8314 tablet_bootstrap.cc:654] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.357218 8315 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:29.357872 8314 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:29.358042 8315 tablet_bootstrap.cc:654] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.358081 8314 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.358534 8314 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.358634 8314 raft_consensus.cc:374] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.358668 8314 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.358780 8314 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.359069 8314 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.359117 8315 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:29.359185 8315 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.359576 8314 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:29.359735 8315 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.359861 8315 raft_consensus.cc:374] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.359894 8315 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.360004 8315 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.360267 8315 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.361760 8314 tablet_bootstrap.cc:654] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.363081 8314 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:29.363154 8314 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:29.363440 8314 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.363512 8314 raft_consensus.cc:374] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.363582 8314 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.363689 8314 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.363907 8314 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.525575 8243 raft_consensus.cc:482] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.525708 8234 raft_consensus.cc:482] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.525748 8243 raft_consensus.cc:504] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.525831 8234 raft_consensus.cc:504] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.526187 8243 leader_election.cc:288] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.526330 8234 leader_election.cc:288] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:29.526796 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:29.526901 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ffb59d3ab64d03aca5d30a7c3be1e1" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:29.526933 7503 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:29.527010 7241 raft_consensus.cc:2454] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:29.527118 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ffb59d3ab64d03aca5d30a7c3be1e1" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:29.527240 7502 raft_consensus.cc:2454] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:29.527240 7177 leader_election.cc:302] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:29.527334 7307 leader_election.cc:302] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:29.527400 8234 raft_consensus.cc:2781] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.527447 8234 raft_consensus.cc:482] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.527494 8234 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.527650 8243 raft_consensus.cc:2781] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.527719 8243 raft_consensus.cc:482] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.527762 8243 raft_consensus.cc:3037] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.528460 8234 raft_consensus.cc:504] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.528578 8243 raft_consensus.cc:504] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.528792 8234 leader_election.cc:288] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:29.528901 8243 leader_election.cc:288] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.529130 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ffb59d3ab64d03aca5d30a7c3be1e1" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:29.529223 7241 raft_consensus.cc:3037] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.529271 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ffb59d3ab64d03aca5d30a7c3be1e1" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:29.529289 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:29.529372 7502 raft_consensus.cc:3037] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.529381 7503 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.530030 7241 raft_consensus.cc:2454] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:29.530282 7502 raft_consensus.cc:2454] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:29.530376 7503 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:29.530550 7308 leader_election.cc:302] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:29.530575 7177 leader_election.cc:302] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:29.530717 8234 raft_consensus.cc:2781] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.530752 8243 raft_consensus.cc:2781] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.530807 8234 raft_consensus.cc:686] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Running, Role: LEADER
I20240503 01:59:29.530841 8243 raft_consensus.cc:686] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:29.530936 8234 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.530956 8243 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:29.531981 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:29.532121 7633 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:29.532123 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:29.532292 7634 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.532954 7090 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:29.533111 7098 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:29.533224 7634 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:29.709270 8283 raft_consensus.cc:482] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.709406 8283 raft_consensus.cc:504] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.710543 8283 leader_election.cc:288] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.714771 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4628afbfeff4f8bbaba5da1e89e4850" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:29.714931 7241 raft_consensus.cc:2454] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:29.715152 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4628afbfeff4f8bbaba5da1e89e4850" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:29.715303 7503 raft_consensus.cc:2454] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:29.715301 7569 leader_election.cc:302] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:29.715561 8283 raft_consensus.cc:2781] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.715629 8283 raft_consensus.cc:482] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.715675 8283 raft_consensus.cc:3037] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.716536 8283 raft_consensus.cc:504] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.716866 8283 leader_election.cc:288] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:29.717213 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4628afbfeff4f8bbaba5da1e89e4850" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:29.717300 7503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4628afbfeff4f8bbaba5da1e89e4850" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:29.717360 7241 raft_consensus.cc:3037] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.717408 7503 raft_consensus.cc:3037] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.718055 7241 raft_consensus.cc:2454] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:29.718062 7503 raft_consensus.cc:2454] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:29.718354 7569 leader_election.cc:302] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:29.718508 8283 raft_consensus.cc:2781] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.718601 8283 raft_consensus.cc:686] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:29.718899 8283 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:29.721263 7090 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:29.907101 8336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:29.907389 8336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:29.910400 8336 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:29.951678 8282 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.956808 8282 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.990620 7503 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.991196 7241 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.991147 8234 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.992051 8283 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.992514 7634 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.993170 8234 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.995041 7503 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.996603 8299 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.525 ms
time per row: 0.0527625 ms
I20240503 01:59:30.056239 7090 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.061290 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 6649e1feab5b40738003119a9122f07a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.061564 7614 tablet_service.cc:1467] Processing CreateTablet for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.061575 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.061887 7483 tablet_service.cc:1467] Processing CreateTablet for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.061987 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 6649e1feab5b40738003119a9122f07a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.062209 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.062242 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.062311 7220 tablet_service.cc:1467] Processing CreateTablet for tablet 6649e1feab5b40738003119a9122f07a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.062553 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.063783 7217 tablet_service.cc:1467] Processing CreateTablet for tablet 5f35ccb9c21b4c10b9187596d2f6929c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.063992 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.064208 8397 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:30.064728 8399 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:30.065039 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.065539 8397 tablet_bootstrap.cc:654] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.065546 8399 tablet_bootstrap.cc:654] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.066020 7350 tablet_service.cc:1467] Processing CreateTablet for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.066242 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.066409 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 5f35ccb9c21b4c10b9187596d2f6929c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.066613 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.067179 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 5f35ccb9c21b4c10b9187596d2f6929c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.067400 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.068809 8398 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:30.068809 8402 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:30.068953 8397 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:30.069072 8397 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:30.069545 8397 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.069691 8398 tablet_bootstrap.cc:654] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.069700 8402 tablet_bootstrap.cc:654] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.069815 8397 raft_consensus.cc:374] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.069900 8397 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.070111 8397 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.070433 8399 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:30.070523 8399 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:30.070533 8397 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.070727 8398 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:30.070793 8398 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.070678 8397 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:30.070938 8399 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.071027 8399 raft_consensus.cc:374] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.071069 8399 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.071161 8398 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.071189 8399 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.071254 8398 raft_consensus.cc:374] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.071283 8398 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.071455 8399 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.071630 8399 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:30.071426 8398 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.072518 8399 tablet_bootstrap.cc:654] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.072530 8398 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.072692 8398 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:30.073654 8398 tablet_bootstrap.cc:654] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.074106 8399 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:30.074189 8399 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.074610 8399 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.074700 8399 raft_consensus.cc:374] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.074740 8399 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.074849 8399 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.075166 8398 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:30.075176 8399 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.075239 8398 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.075685 8398 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.075793 8398 raft_consensus.cc:374] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.075824 8398 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.075860 8402 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:30.075938 8402 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:30.075942 8398 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.076192 8398 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.076349 8398 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:30.076391 8402 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.076478 8402 raft_consensus.cc:374] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.076522 8402 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.076663 8402 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.077005 8402 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.077180 8398 tablet_bootstrap.cc:654] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.075654 8397 tablet_bootstrap.cc:654] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.078200 8398 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:30.078277 8398 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.078722 8398 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.078788 8398 raft_consensus.cc:374] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.078819 8398 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.078912 8398 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.078951 8397 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:30.079031 8397 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 01:59:30.079155 8398 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.079422 8397 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.079525 8397 raft_consensus.cc:374] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.079581 8397 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.079695 8397 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.079954 8397 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.080143 8397 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:30.080945 8397 tablet_bootstrap.cc:654] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.081954 8397 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:30.082046 8397 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:30.082484 8397 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.082583 8397 raft_consensus.cc:374] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.082628 8397 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.082748 8397 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.082962 8397 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.114223 8299 raft_consensus.cc:482] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.114367 8299 raft_consensus.cc:504] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.114835 8299 leader_election.cc:288] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:30.115609 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95d3c29bfbd4899b27b770bf9d3f4a6" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:30.115762 7501 raft_consensus.cc:2454] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:30.116087 7570 leader_election.cc:302] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.116261 8299 raft_consensus.cc:2781] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.116322 8299 raft_consensus.cc:482] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.116364 8299 raft_consensus.cc:3037] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.117460 8299 raft_consensus.cc:504] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.117792 8299 leader_election.cc:288] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:30.118201 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95d3c29bfbd4899b27b770bf9d3f4a6" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:30.118328 7501 raft_consensus.cc:3037] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.118350 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95d3c29bfbd4899b27b770bf9d3f4a6" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:30.118391 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95d3c29bfbd4899b27b770bf9d3f4a6" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:30.118502 7372 raft_consensus.cc:2454] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:30.119169 7501 raft_consensus.cc:2454] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:30.119438 7570 leader_election.cc:302] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a
I20240503 01:59:30.119684 8299 raft_consensus.cc:2781] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.119757 8299 raft_consensus.cc:686] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:30.119899 8299 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.121551 7098 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:30.179514 8390 raft_consensus.cc:482] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.179659 8390 raft_consensus.cc:504] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.180119 8390 leader_election.cc:288] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:30.180555 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f35ccb9c21b4c10b9187596d2f6929c" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:30.180572 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f35ccb9c21b4c10b9187596d2f6929c" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:30.180692 7241 raft_consensus.cc:2454] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:30.180703 7634 raft_consensus.cc:2454] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:30.181025 7307 leader_election.cc:302] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:30.181181 8390 raft_consensus.cc:2781] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.181238 8390 raft_consensus.cc:482] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.181272 8390 raft_consensus.cc:3037] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.182088 8390 raft_consensus.cc:504] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.182406 8390 leader_election.cc:288] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:30.182691 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f35ccb9c21b4c10b9187596d2f6929c" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:30.182734 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f35ccb9c21b4c10b9187596d2f6929c" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:30.182799 7634 raft_consensus.cc:3037] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.182826 7241 raft_consensus.cc:3037] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.183504 7634 raft_consensus.cc:2454] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:30.183595 7241 raft_consensus.cc:2454] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:30.183797 7306 leader_election.cc:302] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.183985 8390 raft_consensus.cc:2781] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.184089 8390 raft_consensus.cc:686] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:30.184238 8390 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:30.186057 7098 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:30.316265 8243 raft_consensus.cc:482] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.316406 8243 raft_consensus.cc:504] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.316833 8243 leader_election.cc:288] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:30.317142 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649e1feab5b40738003119a9122f07a" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:30.317232 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649e1feab5b40738003119a9122f07a" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:30.317301 7633 raft_consensus.cc:2454] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:30.317359 7241 raft_consensus.cc:2454] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:30.317624 7307 leader_election.cc:302] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:30.317795 8243 raft_consensus.cc:2781] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.317847 8243 raft_consensus.cc:482] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.317873 8243 raft_consensus.cc:3037] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.318635 8243 raft_consensus.cc:504] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.318948 8243 leader_election.cc:288] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:30.319238 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649e1feab5b40738003119a9122f07a" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:30.319249 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6649e1feab5b40738003119a9122f07a" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:30.319360 7633 raft_consensus.cc:3037] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.319360 7241 raft_consensus.cc:3037] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.320323 7633 raft_consensus.cc:2454] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:30.320323 7241 raft_consensus.cc:2454] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:30.320569 7306 leader_election.cc:302] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.320706 8243 raft_consensus.cc:2781] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.320817 8243 raft_consensus.cc:686] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:30.320950 8243 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.322631 7098 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:30.472046 8413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.472329 8413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.475337 8413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:30.517746 8299 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.530205 7372 raft_consensus.cc:3037] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.532058 8283 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.560690 7241 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.561064 7633 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.561480 8390 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.562088 8390 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.565227 7634 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.566686 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.567344 8243 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.570592 8390 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.63 ms
time per row: 0.0523151 ms
I20240503 01:59:30.624760 7095 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.629364 7483 tablet_service.cc:1467] Processing CreateTablet for tablet fb00fcf563b74243b7c955ddacf1d6cf (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.629421 7217 tablet_service.cc:1467] Processing CreateTablet for tablet fb00fcf563b74243b7c955ddacf1d6cf (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.629478 7352 tablet_service.cc:1467] Processing CreateTablet for tablet fb00fcf563b74243b7c955ddacf1d6cf (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.629662 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb00fcf563b74243b7c955ddacf1d6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.629676 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb00fcf563b74243b7c955ddacf1d6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.629688 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb00fcf563b74243b7c955ddacf1d6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.630501 7220 tablet_service.cc:1467] Processing CreateTablet for tablet 40260acad62242c3a72197d2dafae187 (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.630693 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.631881 8472 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:30.631880 7350 tablet_service.cc:1467] Processing CreateTablet for tablet be1f4d2622c6452b869f88a301a5e9bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.632054 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.632812 8473 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:30.632839 8472 tablet_bootstrap.cc:654] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.633699 8473 tablet_bootstrap.cc:654] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.634054 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 40260acad62242c3a72197d2dafae187 (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.634141 7482 tablet_service.cc:1467] Processing CreateTablet for tablet be1f4d2622c6452b869f88a301a5e9bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.634287 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.634539 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 40260acad62242c3a72197d2dafae187 (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.634701 8472 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:30.634701 8473 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:30.634771 8473 ts_tablet_manager.cc:1367] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.634771 8472 ts_tablet_manager.cc:1367] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.634788 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.635231 8473 raft_consensus.cc:348] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.635231 8472 raft_consensus.cc:348] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.635330 8473 raft_consensus.cc:374] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.635330 8472 raft_consensus.cc:374] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.635365 8472 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.635365 8473 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.635483 8473 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.635483 8472 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.635758 8473 ts_tablet_manager.cc:1398] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.635758 8472 ts_tablet_manager.cc:1398] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.635808 7614 tablet_service.cc:1467] Processing CreateTablet for tablet be1f4d2622c6452b869f88a301a5e9bc (DEFAULT_TABLE table=rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.635906 8473 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:30.635906 8472 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:30.636044 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.636736 8473 tablet_bootstrap.cc:654] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.636736 8472 tablet_bootstrap.cc:654] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.637785 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.638017 8473 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:30.638088 8473 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.638430 8473 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.638531 8473 raft_consensus.cc:374] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.638559 8473 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.638693 8473 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.639010 8473 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.639533 8472 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:30.639660 8472 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:30.639992 8472 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.640072 8472 raft_consensus.cc:374] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.640092 8472 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.640201 8472 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.640437 8472 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.640789 8478 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:30.641537 8478 tablet_bootstrap.cc:654] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.642149 8471 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:30.642889 8478 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:30.642977 8478 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.643091 8471 tablet_bootstrap.cc:654] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.643384 8478 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.643476 8478 raft_consensus.cc:374] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.643510 8478 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.643762 8478 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.644074 8478 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.644202 8471 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:30.644294 8471 ts_tablet_manager.cc:1367] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.644418 8478 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:30.644721 8471 raft_consensus.cc:348] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.644834 8471 raft_consensus.cc:374] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.644860 8471 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.644968 8471 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.645251 8478 tablet_bootstrap.cc:654] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.645257 8471 ts_tablet_manager.cc:1398] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.645414 8471 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:30.646256 8471 tablet_bootstrap.cc:654] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.646611 8478 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:30.646685 8478 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:30.647069 8478 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.647167 8478 raft_consensus.cc:374] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.647207 8478 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.647346 8478 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.647455 8471 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:30.647526 8471 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.647660 8478 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.647964 8471 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.648066 8471 raft_consensus.cc:374] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.648098 8471 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.648217 8471 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.648449 8471 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.648821 8471 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:30.649658 8471 tablet_bootstrap.cc:654] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.650631 8471 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:30.650692 8471 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.650959 8471 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.651028 8471 raft_consensus.cc:374] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.651064 8471 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.651149 8471 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.651381 8471 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.707217 8460 raft_consensus.cc:482] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.707363 8460 raft_consensus.cc:504] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.707885 8460 leader_election.cc:288] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:30.708343 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1f4d2622c6452b869f88a301a5e9bc" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:30.708343 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1f4d2622c6452b869f88a301a5e9bc" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:30.708488 7501 raft_consensus.cc:2454] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:30.708488 7372 raft_consensus.cc:2454] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:30.708782 7570 leader_election.cc:302] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.708941 8460 raft_consensus.cc:2781] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.708997 8460 raft_consensus.cc:482] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.709044 8460 raft_consensus.cc:3037] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.709811 8460 raft_consensus.cc:504] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.710101 8460 leader_election.cc:288] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:30.710461 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1f4d2622c6452b869f88a301a5e9bc" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:30.710503 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1f4d2622c6452b869f88a301a5e9bc" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:30.710584 7372 raft_consensus.cc:3037] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.710644 7501 raft_consensus.cc:3037] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.711323 7372 raft_consensus.cc:2454] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:30.711408 7501 raft_consensus.cc:2454] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:30.711612 7570 leader_election.cc:302] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.711764 8460 raft_consensus.cc:2781] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.711849 8460 raft_consensus.cc:686] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:30.712013 8460 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:30.713693 7094 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.738925 8359 raft_consensus.cc:482] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.739068 8359 raft_consensus.cc:504] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.740141 8359 leader_election.cc:288] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:30.743927 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40260acad62242c3a72197d2dafae187" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:30.744136 7633 raft_consensus.cc:2454] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:30.744134 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40260acad62242c3a72197d2dafae187" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:30.744268 7240 raft_consensus.cc:2454] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:30.744563 7437 leader_election.cc:302] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:30.744853 8359 raft_consensus.cc:2781] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.744915 8359 raft_consensus.cc:482] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.744963 8359 raft_consensus.cc:3037] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.745730 8359 raft_consensus.cc:504] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.746023 8359 leader_election.cc:288] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:30.746343 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40260acad62242c3a72197d2dafae187" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:30.746368 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40260acad62242c3a72197d2dafae187" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:30.746440 7241 raft_consensus.cc:3037] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.746488 7633 raft_consensus.cc:3037] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.747104 7241 raft_consensus.cc:2454] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:30.747263 7633 raft_consensus.cc:2454] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:30.747368 7438 leader_election.cc:302] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:30.747529 8359 raft_consensus.cc:2781] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.747743 8359 raft_consensus.cc:686] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:30.748044 8359 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:30.750213 7094 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:30.834954 8359 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.835103 8359 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.835598 8359 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:30.836091 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:30.836268 7241 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:30.836535 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:30.836699 8359 raft_consensus.cc:2781] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.836788 8359 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.836820 8359 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.837697 8359 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.838058 8359 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:30.838423 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:30.838541 7241 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.839164 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:30.839138 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:30.839278 7371 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.839392 7241 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:30.839697 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:30.839855 8359 raft_consensus.cc:2781] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.839949 8359 raft_consensus.cc:686] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:30.840092 8359 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:30.840739 7371 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:30.841943 7094 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:30.981727 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.982010 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.984896 8491 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:31.071745 7241 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.071745 7240 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.072259 8359 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.072640 8359 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.074784 7371 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.076396 8488 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.082056 7633 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.082576 7372 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.082924 8460 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.083441 8488 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.083501 7501 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.086122 8459 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.783 ms
time per row: 0.0578915 ms
I20240503 01:59:31.142586 7096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:31.148034 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 0d00b8fcb29f47298643903ec0cba606 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.148094 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 0d00b8fcb29f47298643903ec0cba606 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.148398 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d00b8fcb29f47298643903ec0cba606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.148401 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d00b8fcb29f47298643903ec0cba606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.149152 7220 tablet_service.cc:1467] Processing CreateTablet for tablet 0d00b8fcb29f47298643903ec0cba606 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.149441 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d00b8fcb29f47298643903ec0cba606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.150527 7217 tablet_service.cc:1467] Processing CreateTablet for tablet a432cd5fe20941d6bc73dae386d85170 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.150729 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.150873 8533 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:31.151693 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.151769 8533 tablet_bootstrap.cc:654] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.151919 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.152518 8471 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:31.152776 8533 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:31.152854 8533 ts_tablet_manager.cc:1367] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.153242 8471 tablet_bootstrap.cc:654] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.153287 8533 raft_consensus.cc:348] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.153395 8533 raft_consensus.cc:374] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.153432 8533 raft_consensus.cc:729] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.153553 8533 consensus_queue.cc:260] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.153810 8533 ts_tablet_manager.cc:1398] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.154683 8471 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:31.154765 8471 ts_tablet_manager.cc:1367] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.154786 7613 tablet_service.cc:1467] Processing CreateTablet for tablet a432cd5fe20941d6bc73dae386d85170 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.155097 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.155252 8471 raft_consensus.cc:348] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.155345 8471 raft_consensus.cc:374] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.155377 8471 raft_consensus.cc:729] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.155493 8471 consensus_queue.cc:260] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.155798 8471 ts_tablet_manager.cc:1398] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.155791 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.156013 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.156642 7350 tablet_service.cc:1467] Processing CreateTablet for tablet a432cd5fe20941d6bc73dae386d85170 (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.156845 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.157759 8534 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:31.158607 8534 tablet_bootstrap.cc:654] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.158694 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e (DEFAULT_TABLE table=rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.158864 8533 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:31.158938 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.159610 8533 tablet_bootstrap.cc:654] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.160513 8533 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:31.160589 8533 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:31.160967 8533 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.161062 8533 raft_consensus.cc:374] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.161089 8533 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.161098 8466 raft_consensus.cc:482] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.161190 8466 raft_consensus.cc:504] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.161212 8533 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.161489 8533 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.161594 8466 leader_election.cc:288] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.162011 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d00b8fcb29f47298643903ec0cba606" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:31.162431 8534 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:31.162515 8534 ts_tablet_manager.cc:1367] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:31.162855 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d00b8fcb29f47298643903ec0cba606" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:31.162997 7501 raft_consensus.cc:2454] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:31.163499 7308 leader_election.cc:302] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:31.163677 8466 raft_consensus.cc:2781] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.163733 8466 raft_consensus.cc:482] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.163769 8466 raft_consensus.cc:3037] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:31.163789 7307 leader_election.cc:341] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:31.163877 8537 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:31.164260 8534 raft_consensus.cc:348] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.164362 8534 raft_consensus.cc:374] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.164393 8534 raft_consensus.cc:729] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.164558 8534 consensus_queue.cc:260] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.164656 8466 raft_consensus.cc:504] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.164990 8466 leader_election.cc:288] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.165023 8534 ts_tablet_manager.cc:1398] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.165156 8471 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:31.165174 8534 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:31.165335 8537 tablet_bootstrap.cc:654] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.165347 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d00b8fcb29f47298643903ec0cba606" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:31.165434 7240 raft_consensus.cc:3037] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.165997 8534 tablet_bootstrap.cc:654] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.165999 8471 tablet_bootstrap.cc:654] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.166419 7240 raft_consensus.cc:2454] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:31.166819 7307 leader_election.cc:302] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:31.167057 8534 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:31.167106 8466 raft_consensus.cc:2781] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.167133 8534 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.167101 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d00b8fcb29f47298643903ec0cba606" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:31.167189 8466 raft_consensus.cc:686] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:31.167193 7501 raft_consensus.cc:3037] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.167308 8466 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.167582 8534 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.167681 8534 raft_consensus.cc:374] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.167711 8534 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.167810 8534 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.168123 8534 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.168123 7501 raft_consensus.cc:2454] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:31.168285 8534 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:31.169091 8534 tablet_bootstrap.cc:654] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.169900 8537 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:31.169994 8537 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:31.170075 8534 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:31.170142 8534 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.170419 8537 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.170521 8537 raft_consensus.cc:374] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.170554 8537 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.170542 8534 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.170644 8534 raft_consensus.cc:374] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.170675 8534 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.170759 8537 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.170790 8534 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.171031 8534 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.171046 8537 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.171221 8537 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:31.171142 7091 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:31.171989 8537 tablet_bootstrap.cc:654] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.173393 8537 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:31.173398 8471 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:31.173470 8537 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:31.173482 8471 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:31.173823 8537 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.173942 8537 raft_consensus.cc:374] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.173938 8471 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.174012 8537 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.174044 8471 raft_consensus.cc:374] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.174082 8471 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.174126 8537 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.174216 8471 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.174479 8537 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.174938 8471 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.236927 8488 raft_consensus.cc:482] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.237071 8488 raft_consensus.cc:504] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.237577 8488 leader_election.cc:288] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.238040 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebe541161c84bd6b11bcd3d13d3bd2e" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:31.238041 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebe541161c84bd6b11bcd3d13d3bd2e" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:31.238170 7633 raft_consensus.cc:2454] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:31.238204 7240 raft_consensus.cc:2454] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:31.238509 7437 leader_election.cc:302] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:31.238693 8488 raft_consensus.cc:2781] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.238759 8488 raft_consensus.cc:482] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.238819 8488 raft_consensus.cc:3037] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.239769 8488 raft_consensus.cc:504] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.240139 8488 leader_election.cc:288] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.240444 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebe541161c84bd6b11bcd3d13d3bd2e" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:31.240458 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebe541161c84bd6b11bcd3d13d3bd2e" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:31.240572 7240 raft_consensus.cc:3037] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.240571 7633 raft_consensus.cc:3037] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.241345 7240 raft_consensus.cc:2454] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:31.241364 7633 raft_consensus.cc:2454] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:31.241616 7437 leader_election.cc:302] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:31.241787 8488 raft_consensus.cc:2781] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.241870 8488 raft_consensus.cc:686] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:31.242027 8488 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.243691 7091 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:31.442906 8243 raft_consensus.cc:482] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.443035 8243 raft_consensus.cc:504] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.443388 8243 leader_election.cc:288] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.443775 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a432cd5fe20941d6bc73dae386d85170" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:31.443774 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a432cd5fe20941d6bc73dae386d85170" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:31.443907 7240 raft_consensus.cc:2454] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:31.443920 7633 raft_consensus.cc:2454] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:31.444201 7306 leader_election.cc:302] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:31.444368 8243 raft_consensus.cc:2781] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.444422 8243 raft_consensus.cc:482] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.444482 8243 raft_consensus.cc:3037] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.445333 8243 raft_consensus.cc:504] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.445662 8243 leader_election.cc:288] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:31.446017 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a432cd5fe20941d6bc73dae386d85170" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:31.446049 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a432cd5fe20941d6bc73dae386d85170" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:31.446161 7240 raft_consensus.cc:3037] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.446178 7633 raft_consensus.cc:3037] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.447032 7633 raft_consensus.cc:2454] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:31.447032 7240 raft_consensus.cc:2454] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:31.447284 7307 leader_election.cc:302] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:31.447484 8243 raft_consensus.cc:2781] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.447597 8243 raft_consensus.cc:686] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:31.447764 8243 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:31.449378 7091 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:31.557565 8548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.557813 8548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.560658 8548 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:31.645807 7240 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.645953 7633 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.646215 8466 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.646339 8243 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.648911 8466 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.652612 7239 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.652993 8488 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.654145 7502 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.655225 8466 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.656984 7634 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.658324 8488 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.061 ms
time per row: 0.0620305 ms
I20240503 01:59:31.721776 7092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:31.726769 7482 tablet_service.cc:1467] Processing CreateTablet for tablet f0539dded7554723b9e918e8cdfbab20 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.726902 7613 tablet_service.cc:1467] Processing CreateTablet for tablet f0539dded7554723b9e918e8cdfbab20 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.727070 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.727123 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.727301 7350 tablet_service.cc:1467] Processing CreateTablet for tablet f0539dded7554723b9e918e8cdfbab20 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.727526 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.729151 8603 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:31.729318 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 3b15d48b2fa04269b98c6f801f54df4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.729487 8605 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:31.729542 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b15d48b2fa04269b98c6f801f54df4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.729591 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 245cba4e5728491896e7995ca0d59b36 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.729830 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245cba4e5728491896e7995ca0d59b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.730088 8603 tablet_bootstrap.cc:654] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.730377 8605 tablet_bootstrap.cc:654] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.731428 8605 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:31.731423 8604 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:31.731499 8605 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.732164 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 3b15d48b2fa04269b98c6f801f54df4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.732316 8604 tablet_bootstrap.cc:654] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.732288 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 245cba4e5728491896e7995ca0d59b36 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.732409 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b15d48b2fa04269b98c6f801f54df4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.732518 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245cba4e5728491896e7995ca0d59b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.732791 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 245cba4e5728491896e7995ca0d59b36 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.733028 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245cba4e5728491896e7995ca0d59b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.734025 8605 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.734131 8605 raft_consensus.cc:374] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.734202 8605 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.734339 8605 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.734586 8605 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:31.734722 8605 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:31.735302 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 3b15d48b2fa04269b98c6f801f54df4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.735498 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b15d48b2fa04269b98c6f801f54df4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.735847 8605 tablet_bootstrap.cc:654] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.736883 8605 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:31.736951 8605 ts_tablet_manager.cc:1367] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.737357 8605 raft_consensus.cc:348] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.737448 8605 raft_consensus.cc:374] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.737495 8605 raft_consensus.cc:729] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.737520 8610 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:31.737613 8605 consensus_queue.cc:260] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.737846 8605 ts_tablet_manager.cc:1398] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.738034 8605 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:31.738281 8604 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:31.738348 8610 tablet_bootstrap.cc:654] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.738358 8604 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:31.738749 8604 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.738853 8604 raft_consensus.cc:374] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.738889 8604 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.738906 8605 tablet_bootstrap.cc:654] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.739116 8604 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.739343 8603 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:31.739423 8603 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 01:59:31.739436 8604 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.739605 8604 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:31.739840 8603 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.739930 8603 raft_consensus.cc:374] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.739961 8603 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.740067 8603 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.740303 8603 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.740334 8604 tablet_bootstrap.cc:654] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.740443 8603 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:31.741003 8610 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:31.741071 8610 ts_tablet_manager.cc:1367] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:31.741250 8603 tablet_bootstrap.cc:654] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.741252 8604 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:31.741343 8604 ts_tablet_manager.cc:1367] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.741528 8610 raft_consensus.cc:348] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.741621 8610 raft_consensus.cc:374] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.741652 8610 raft_consensus.cc:729] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.741698 8604 raft_consensus.cc:348] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.741786 8604 raft_consensus.cc:374] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.741815 8604 raft_consensus.cc:729] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.741796 8610 consensus_queue.cc:260] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.741931 8604 consensus_queue.cc:260] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.742182 8604 ts_tablet_manager.cc:1398] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.742535 8610 ts_tablet_manager.cc:1398] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.742872 8603 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:31.742946 8603 ts_tablet_manager.cc:1367] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:31.743148 8605 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:31.743213 8605 ts_tablet_manager.cc:1367] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:31.743389 8603 raft_consensus.cc:348] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.743484 8603 raft_consensus.cc:374] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.743520 8603 raft_consensus.cc:729] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.743613 8605 raft_consensus.cc:348] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.743642 8603 consensus_queue.cc:260] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.743698 8605 raft_consensus.cc:374] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.743733 8605 raft_consensus.cc:729] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.743831 8605 consensus_queue.cc:260] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:31.743880 8603 ts_tablet_manager.cc:1398] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.744037 8603 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:31.744644 8605 ts_tablet_manager.cc:1398] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.744892 8603 tablet_bootstrap.cc:654] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.746261 8603 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:31.746364 8603 ts_tablet_manager.cc:1367] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:31.746722 8603 raft_consensus.cc:348] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.746826 8603 raft_consensus.cc:374] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.746863 8603 raft_consensus.cc:729] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.746976 8603 consensus_queue.cc:260] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.747264 8603 ts_tablet_manager.cc:1398] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.819223 8488 raft_consensus.cc:482] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.819350 8488 raft_consensus.cc:504] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.819789 8488 leader_election.cc:288] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:31.820204 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b15d48b2fa04269b98c6f801f54df4f" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:31.820204 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b15d48b2fa04269b98c6f801f54df4f" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:31.820372 7634 raft_consensus.cc:2454] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:31.820372 7372 raft_consensus.cc:2454] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:31.820657 7437 leader_election.cc:302] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:31.820832 8488 raft_consensus.cc:2781] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.820914 8488 raft_consensus.cc:482] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.820968 8488 raft_consensus.cc:3037] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.821707 8488 raft_consensus.cc:504] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.822019 8488 leader_election.cc:288] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:31.822315 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b15d48b2fa04269b98c6f801f54df4f" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:31.822422 7634 raft_consensus.cc:3037] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.822391 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b15d48b2fa04269b98c6f801f54df4f" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:31.822510 7372 raft_consensus.cc:3037] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.823197 7634 raft_consensus.cc:2454] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:31.823385 7372 raft_consensus.cc:2454] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:31.823637 7437 leader_election.cc:302] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:31.823957 8488 raft_consensus.cc:2781] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.824051 8488 raft_consensus.cc:686] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:31.824184 8488 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:31.825770 7091 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.005270 8488 raft_consensus.cc:482] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.005399 8488 raft_consensus.cc:504] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.005769 8488 leader_election.cc:288] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.005950 8591 raft_consensus.cc:482] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.006103 8591 raft_consensus.cc:504] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.006323 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.006459 7239 raft_consensus.cc:2454] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:32.006506 8591 leader_election.cc:288] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.006680 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:32.006788 7372 raft_consensus.cc:2454] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:32.006801 7438 leader_election.cc:302] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:32.006820 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.006850 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:32.006923 7239 raft_consensus.cc:2454] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.006970 7502 raft_consensus.cc:2454] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.007042 8488 raft_consensus.cc:2781] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.007091 8488 raft_consensus.cc:482] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.007149 8488 raft_consensus.cc:3037] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.007193 7307 leader_election.cc:302] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:32.007364 8591 raft_consensus.cc:2781] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.007423 8591 raft_consensus.cc:482] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.007467 8591 raft_consensus.cc:3037] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.008060 8488 raft_consensus.cc:504] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.008316 8591 raft_consensus.cc:504] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.008599 8591 leader_election.cc:288] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.008630 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.008630 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:32.008724 7239 raft_consensus.cc:3037] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.008777 7372 raft_consensus.cc:2379] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in current term 1: Already voted for candidate bed387c0ff9241688cdf45f364dc6f5a in this term.
I20240503 01:59:32.008896 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.008961 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245cba4e5728491896e7995ca0d59b36" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:32.009074 7502 raft_consensus.cc:2379] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bed387c0ff9241688cdf45f364dc6f5a in current term 1: Already voted for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in this term.
I20240503 01:59:32.009133 8488 leader_election.cc:288] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.009372 8459 raft_consensus.cc:482] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.009454 7308 leader_election.cc:302] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a; no voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 01:59:32.009477 8459 raft_consensus.cc:504] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.009639 8591 raft_consensus.cc:2726] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 01:59:32.009716 7239 raft_consensus.cc:2454] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:32.009879 8459 leader_election.cc:288] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.009965 7438 leader_election.cc:302] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: bed387c0ff9241688cdf45f364dc6f5a
I20240503 01:59:32.010146 8488 raft_consensus.cc:2781] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.010147 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0539dded7554723b9e918e8cdfbab20" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:32.010229 8488 raft_consensus.cc:686] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:32.010246 7372 raft_consensus.cc:2454] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.010417 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0539dded7554723b9e918e8cdfbab20" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:32.010394 8488 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.010499 7502 raft_consensus.cc:2454] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.010521 7570 leader_election.cc:302] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.010661 8459 raft_consensus.cc:2781] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.010727 8459 raft_consensus.cc:482] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.010784 8459 raft_consensus.cc:3037] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.011679 8459 raft_consensus.cc:504] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.012001 8459 leader_election.cc:288] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.012272 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0539dded7554723b9e918e8cdfbab20" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:32.012369 7372 raft_consensus.cc:3037] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.012352 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0539dded7554723b9e918e8cdfbab20" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:32.012262 7091 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.012437 7502 raft_consensus.cc:3037] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.013257 7372 raft_consensus.cc:2454] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.013317 7502 raft_consensus.cc:2454] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.013499 7570 leader_election.cc:302] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.013654 8459 raft_consensus.cc:2781] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.013859 8459 raft_consensus.cc:686] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:32.013993 8459 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.015609 7091 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:32.137099 8620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.137434 8620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.141790 8620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 01:59:32.223868 7372 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.224347 8520 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.225682 7239 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.226419 7634 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.227326 8520 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.227638 8520 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.230126 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.230479 7370 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.230542 7371 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.230978 8618 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.231706 8618 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.233780 8488 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.250360 8657 mvcc.cc:205] Tried to move back new op lower bound from 7023417639831343104 to 7023417638956036096. Current Snapshot: MvccSnapshot[applied={T|T < 7023417639831343104}]
INSERT report
rows total: 2000
time total: 103.626 ms
time per row: 0.0518132 ms
I20240503 01:59:32.293372 7091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.299000 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.299096 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.299351 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.299351 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.299949 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.299968 7352 tablet_service.cc:1467] Processing CreateTablet for tablet ec179e5e683d4ba099c459331ea5e578 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.300354 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.300508 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.301468 7217 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.301710 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.302183 7220 tablet_service.cc:1467] Processing CreateTablet for tablet ec179e5e683d4ba099c459331ea5e578 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.302392 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.302333 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.302845 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.303357 8679 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:32.304776 8679 tablet_bootstrap.cc:654] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.305182 8681 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:32.305670 8680 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:32.305625 7613 tablet_service.cc:1467] Processing CreateTablet for tablet ec179e5e683d4ba099c459331ea5e578 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.305763 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 (DEFAULT_TABLE table=rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.305861 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.306008 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.306567 8680 tablet_bootstrap.cc:654] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.307147 8681 tablet_bootstrap.cc:654] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.308298 8681 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:32.308377 8681 ts_tablet_manager.cc:1367] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.308523 8679 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:32.308605 8679 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:32.308884 8681 raft_consensus.cc:348] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.308996 8681 raft_consensus.cc:374] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.309033 8681 raft_consensus.cc:729] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.309019 8679 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.309125 8679 raft_consensus.cc:374] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.309161 8679 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.309185 8681 consensus_queue.cc:260] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.309330 8679 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.309623 8679 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.309816 8679 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:32.310068 8681 ts_tablet_manager.cc:1398] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.310866 8679 tablet_bootstrap.cc:654] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.311959 8679 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:32.312042 8679 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:32.312042 8680 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:32.312139 8680 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:32.312546 8680 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.312530 8679 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.312628 8680 raft_consensus.cc:374] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.312633 8679 raft_consensus.cc:374] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.312652 8680 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.312677 8679 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.312801 8680 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.312799 8679 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.313128 8680 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.313244 8679 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.313282 8680 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:32.313405 8679 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:32.314145 8680 tablet_bootstrap.cc:654] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.314340 8679 tablet_bootstrap.cc:654] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.315389 8680 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:32.315483 8680 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.315968 8680 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.316062 8680 raft_consensus.cc:374] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.316097 8680 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.316201 8680 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.316469 8680 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.317107 8679 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:32.317199 8679 ts_tablet_manager.cc:1367] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:32.317617 8679 raft_consensus.cc:348] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.317705 8679 raft_consensus.cc:374] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.317725 8679 raft_consensus.cc:729] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.317826 8679 consensus_queue.cc:260] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.318059 8679 ts_tablet_manager.cc:1398] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.321471 8688 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:32.322327 8688 tablet_bootstrap.cc:654] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.323585 8688 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:32.323681 8688 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.324084 8688 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.324517 8688 raft_consensus.cc:374] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.324568 8688 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.324996 8688 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.325248 8688 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.325424 8688 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:32.326081 8688 tablet_bootstrap.cc:654] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.326975 8688 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:32.327052 8688 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.327433 8688 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.327566 8688 raft_consensus.cc:374] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.327605 8688 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.327757 8688 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.328061 8688 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.328224 8688 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:32.329069 8688 tablet_bootstrap.cc:654] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.330072 8688 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:32.330147 8688 ts_tablet_manager.cc:1367] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.330555 8688 raft_consensus.cc:348] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.330633 8688 raft_consensus.cc:374] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.330682 8688 raft_consensus.cc:729] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.330812 8688 consensus_queue.cc:260] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.331040 8688 ts_tablet_manager.cc:1398] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.348138 8618 raft_consensus.cc:482] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.348285 8618 raft_consensus.cc:504] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.348770 8618 leader_election.cc:288] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.349161 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba04b45ae1343f3aa4e5a82bbf7f486" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.349180 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba04b45ae1343f3aa4e5a82bbf7f486" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:32.349313 7502 raft_consensus.cc:2454] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.349313 7239 raft_consensus.cc:2454] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.349632 7569 leader_election.cc:302] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.349848 8618 raft_consensus.cc:2781] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.349915 8618 raft_consensus.cc:482] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.349964 8618 raft_consensus.cc:3037] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.350777 8618 raft_consensus.cc:504] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.351106 8618 leader_election.cc:288] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:32.351476 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba04b45ae1343f3aa4e5a82bbf7f486" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:32.351477 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba04b45ae1343f3aa4e5a82bbf7f486" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.351621 7502 raft_consensus.cc:3037] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.351621 7239 raft_consensus.cc:3037] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.352686 7502 raft_consensus.cc:2454] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.353006 7570 leader_election.cc:302] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.353021 7239 raft_consensus.cc:2454] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.353184 8618 raft_consensus.cc:2781] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.353305 8618 raft_consensus.cc:686] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:32.353453 8618 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:32.355278 7093 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.455449 8618 raft_consensus.cc:482] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.455613 8618 raft_consensus.cc:504] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.456118 8618 leader_election.cc:288] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.456611 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec179e5e683d4ba099c459331ea5e578" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:32.456630 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec179e5e683d4ba099c459331ea5e578" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.456746 7370 raft_consensus.cc:2454] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.456805 7239 raft_consensus.cc:2454] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.457046 7570 leader_election.cc:302] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.457237 8618 raft_consensus.cc:2781] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.457291 8618 raft_consensus.cc:482] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.457341 8618 raft_consensus.cc:3037] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.458181 8618 raft_consensus.cc:504] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.458498 8618 leader_election.cc:288] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.458784 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec179e5e683d4ba099c459331ea5e578" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.458792 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec179e5e683d4ba099c459331ea5e578" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:32.458923 7239 raft_consensus.cc:3037] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.458923 7370 raft_consensus.cc:3037] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.459873 7239 raft_consensus.cc:2454] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.459890 7370 raft_consensus.cc:2454] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.460119 7570 leader_election.cc:302] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.460270 8618 raft_consensus.cc:2781] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.460348 8618 raft_consensus.cc:686] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:32.460497 8618 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.462570 7093 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.474368 8591 raft_consensus.cc:482] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.474483 8591 raft_consensus.cc:504] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.474869 8591 leader_election.cc:288] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.475190 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad24b47690b4263bdf4a9157db7c17f" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:32.475251 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad24b47690b4263bdf4a9157db7c17f" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.475306 7634 raft_consensus.cc:2454] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.475397 7239 raft_consensus.cc:2454] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.475648 7307 leader_election.cc:302] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:32.475814 8591 raft_consensus.cc:2781] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.475878 8591 raft_consensus.cc:482] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.475905 8591 raft_consensus.cc:3037] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.476668 8591 raft_consensus.cc:504] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.476951 8591 leader_election.cc:288] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.477226 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad24b47690b4263bdf4a9157db7c17f" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:32.477229 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad24b47690b4263bdf4a9157db7c17f" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.477319 7634 raft_consensus.cc:3037] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.477327 7239 raft_consensus.cc:3037] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.478025 7634 raft_consensus.cc:2454] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:32.478058 7239 raft_consensus.cc:2454] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:32.478317 7306 leader_election.cc:302] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.478478 8591 raft_consensus.cc:2781] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.478674 8591 raft_consensus.cc:686] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:32.478790 8591 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.480384 7093 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:32.597769 8696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.597987 8696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.600817 8696 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 01:59:32.703325 7634 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.704191 8694 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.704308 7502 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.704427 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.704814 7241 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.705127 8694 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.705389 8459 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.705597 8618 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.709403 7240 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.709838 8459 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.711493 7370 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.713847 8618 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.724267 8659 mvcc.cc:205] Tried to move back new op lower bound from 7023417641803427840 to 7023417640799666176. Current Snapshot: MvccSnapshot[applied={T|T < 7023417641803427840}]
I20240503 01:59:32.727008 8655 mvcc.cc:205] Tried to move back new op lower bound from 7023417641803427840 to 7023417640799666176. Current Snapshot: MvccSnapshot[applied={T|T < 7023417641803427840}]
INSERT report
rows total: 2000
time total: 136.723 ms
time per row: 0.0683617 ms
I20240503 01:59:32.784533 7092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.790904 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 7d332a7c255b44589d54b1f583a2f9c4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.791276 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d332a7c255b44589d54b1f583a2f9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.792100 7217 tablet_service.cc:1467] Processing CreateTablet for tablet 7d332a7c255b44589d54b1f583a2f9c4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.792387 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d332a7c255b44589d54b1f583a2f9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.792392 7613 tablet_service.cc:1467] Processing CreateTablet for tablet cc68f31e6ef8411e96d46b2419bf9676 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.792651 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc68f31e6ef8411e96d46b2419bf9676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.793747 8681 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:32.794677 8679 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:32.794704 8681 tablet_bootstrap.cc:654] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.795212 8688 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:32.794898 7614 tablet_service.cc:1467] Processing CreateTablet for tablet c18129344d124a5f8f688ec35ef69a55 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.795836 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 7d332a7c255b44589d54b1f583a2f9c4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.795969 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18129344d124a5f8f688ec35ef69a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.796041 7482 tablet_service.cc:1467] Processing CreateTablet for tablet cc68f31e6ef8411e96d46b2419bf9676 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.796104 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d332a7c255b44589d54b1f583a2f9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.796258 8679 tablet_bootstrap.cc:654] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.796315 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc68f31e6ef8411e96d46b2419bf9676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.798321 7350 tablet_service.cc:1467] Processing CreateTablet for tablet cc68f31e6ef8411e96d46b2419bf9676 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.798492 8688 tablet_bootstrap.cc:654] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.798549 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc68f31e6ef8411e96d46b2419bf9676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.798915 7483 tablet_service.cc:1467] Processing CreateTablet for tablet c18129344d124a5f8f688ec35ef69a55 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.799151 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18129344d124a5f8f688ec35ef69a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.799750 8680 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:32.800581 8680 tablet_bootstrap.cc:654] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.800683 7217 tablet_service.cc:1467] Processing CreateTablet for tablet c18129344d124a5f8f688ec35ef69a55 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.800931 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18129344d124a5f8f688ec35ef69a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.801626 8681 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:32.801725 8681 ts_tablet_manager.cc:1367] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:32.802263 8681 raft_consensus.cc:348] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.802398 8681 raft_consensus.cc:374] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.802443 8681 raft_consensus.cc:729] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.802588 8681 consensus_queue.cc:260] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.802909 8681 ts_tablet_manager.cc:1398] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:32.803856 8688 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:32.803941 8688 ts_tablet_manager.cc:1367] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:32.804281 8681 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:32.804641 8679 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:32.804688 8688 raft_consensus.cc:348] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.804739 8679 ts_tablet_manager.cc:1367] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:32.804788 8688 raft_consensus.cc:374] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.804818 8688 raft_consensus.cc:729] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.804953 8688 consensus_queue.cc:260] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.805197 8681 tablet_bootstrap.cc:654] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.805233 8688 ts_tablet_manager.cc:1398] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.805297 8679 raft_consensus.cc:348] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.805395 8688 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:32.805425 8679 raft_consensus.cc:374] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.805462 8679 raft_consensus.cc:729] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.805603 8679 consensus_queue.cc:260] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.805924 8679 ts_tablet_manager.cc:1398] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.806102 8679 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:32.806237 8688 tablet_bootstrap.cc:654] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.807252 8679 tablet_bootstrap.cc:654] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.807305 8688 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:32.807375 8688 ts_tablet_manager.cc:1367] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.807826 8688 raft_consensus.cc:348] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.807955 8688 raft_consensus.cc:374] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.807988 8688 raft_consensus.cc:729] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.808095 8680 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:32.808123 8688 consensus_queue.cc:260] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.808208 8680 ts_tablet_manager.cc:1367] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:32.808378 8688 ts_tablet_manager.cc:1398] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.808670 8680 raft_consensus.cc:348] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.808775 8680 raft_consensus.cc:374] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.808804 8680 raft_consensus.cc:729] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.808915 8680 consensus_queue.cc:260] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.809221 8680 ts_tablet_manager.cc:1398] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.809481 8681 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:32.809568 8681 ts_tablet_manager.cc:1367] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:32.810046 8681 raft_consensus.cc:348] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.810151 8681 raft_consensus.cc:374] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.810181 8681 raft_consensus.cc:729] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.810282 8681 consensus_queue.cc:260] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.810557 8681 ts_tablet_manager.cc:1398] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.810730 8681 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:32.810992 8679 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:32.811072 8679 ts_tablet_manager.cc:1367] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:32.811712 8681 tablet_bootstrap.cc:654] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.811995 8680 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:32.811476 8679 raft_consensus.cc:348] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.812727 8679 raft_consensus.cc:374] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.812790 8679 raft_consensus.cc:729] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.812944 8681 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:32.812937 8679 consensus_queue.cc:260] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.813030 8681 ts_tablet_manager.cc:1367] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.813262 8679 ts_tablet_manager.cc:1398] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:32.813488 8681 raft_consensus.cc:348] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.813596 8681 raft_consensus.cc:374] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.813642 8681 raft_consensus.cc:729] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.813762 8681 consensus_queue.cc:260] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.814110 8681 ts_tablet_manager.cc:1398] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.814141 8680 tablet_bootstrap.cc:654] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.815872 8680 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:32.816000 8680 ts_tablet_manager.cc:1367] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:32.816669 8680 raft_consensus.cc:348] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.816825 8680 raft_consensus.cc:374] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.816857 8680 raft_consensus.cc:729] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.816988 8680 consensus_queue.cc:260] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.817319 8680 ts_tablet_manager.cc:1398] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:32.853721 8664 raft_consensus.cc:482] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.853880 8664 raft_consensus.cc:504] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.854383 8664 leader_election.cc:288] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.854882 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18129344d124a5f8f688ec35ef69a55" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.854893 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18129344d124a5f8f688ec35ef69a55" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:32.855028 7240 raft_consensus.cc:2454] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:32.855070 7634 raft_consensus.cc:2454] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:32.855365 7438 leader_election.cc:302] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:32.855603 8664 raft_consensus.cc:2781] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.855669 8664 raft_consensus.cc:482] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.855716 8664 raft_consensus.cc:3037] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.857028 8664 raft_consensus.cc:504] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.857405 8664 leader_election.cc:288] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.857758 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18129344d124a5f8f688ec35ef69a55" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:32.857901 7634 raft_consensus.cc:3037] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.857913 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18129344d124a5f8f688ec35ef69a55" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.858036 7240 raft_consensus.cc:3037] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.858935 7634 raft_consensus.cc:2454] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:32.858996 7240 raft_consensus.cc:2454] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:32.859261 7437 leader_election.cc:302] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.859472 8664 raft_consensus.cc:2781] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.859681 8664 raft_consensus.cc:686] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:32.859853 8664 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:32.861661 7095 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.934664 8694 raft_consensus.cc:482] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.934849 8694 raft_consensus.cc:504] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.935309 8694 leader_election.cc:288] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.935727 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d332a7c255b44589d54b1f583a2f9c4" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:32.935787 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d332a7c255b44589d54b1f583a2f9c4" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:32.935870 7502 raft_consensus.cc:2454] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.935971 7240 raft_consensus.cc:2454] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:32.936170 7308 leader_election.cc:302] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:32.936368 8694 raft_consensus.cc:2781] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.936424 8694 raft_consensus.cc:482] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.936448 8694 raft_consensus.cc:3037] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.937276 8694 raft_consensus.cc:504] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.937574 8694 leader_election.cc:288] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:32.937937 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d332a7c255b44589d54b1f583a2f9c4" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:32.937942 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d332a7c255b44589d54b1f583a2f9c4" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:32.938069 7502 raft_consensus.cc:3037] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.938093 7240 raft_consensus.cc:3037] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.938838 7502 raft_consensus.cc:2454] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:32.938845 7240 raft_consensus.cc:2454] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:32.939119 7308 leader_election.cc:302] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:32.939337 8694 raft_consensus.cc:2781] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.939491 8694 raft_consensus.cc:686] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:32.939783 8694 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:32.942615 7095 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.970587 8731 raft_consensus.cc:482] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.970750 8731 raft_consensus.cc:504] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.971197 8731 leader_election.cc:288] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.971693 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc68f31e6ef8411e96d46b2419bf9676" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:32.971762 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc68f31e6ef8411e96d46b2419bf9676" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:32.971832 7502 raft_consensus.cc:2454] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.971966 7370 raft_consensus.cc:2454] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 0.
I20240503 01:59:32.972244 7570 leader_election.cc:302] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.972421 8731 raft_consensus.cc:2781] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.972483 8731 raft_consensus.cc:482] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.972533 8731 raft_consensus.cc:3037] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.973419 8731 raft_consensus.cc:504] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.973750 8731 leader_election.cc:288] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:32.974134 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc68f31e6ef8411e96d46b2419bf9676" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:32.974159 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc68f31e6ef8411e96d46b2419bf9676" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:32.974244 7502 raft_consensus.cc:3037] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.974272 7370 raft_consensus.cc:3037] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.975037 7502 raft_consensus.cc:2454] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.975106 7370 raft_consensus.cc:2454] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 1.
I20240503 01:59:32.975400 7570 leader_election.cc:302] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:32.975601 8731 raft_consensus.cc:2781] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.975705 8731 raft_consensus.cc:686] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:32.975858 8731 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:32.977651 7095 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:33.107789 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.108067 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.111166 8746 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 01:59:33.190084 7370 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.190600 8731 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.191952 7502 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.192396 8731 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.201223 7240 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.201972 7501 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.202023 7240 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.202423 8591 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.202692 8729 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.203626 8742 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.204741 7634 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.208586 8650 mvcc.cc:205] Tried to move back new op lower bound from 7023417643781533696 to 7023417642910593024. Current Snapshot: MvccSnapshot[applied={T|T < 7023417643781533696}]
I20240503 01:59:33.212934 8664 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.724 ms
time per row: 0.0553618 ms
I20240503 01:59:33.267155 7095 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.272835 7350 tablet_service.cc:1467] Processing CreateTablet for tablet a0a526f7fced4a04b4e0327f4656cf69 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.272903 7483 tablet_service.cc:1467] Processing CreateTablet for tablet a0a526f7fced4a04b4e0327f4656cf69 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.273200 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0a526f7fced4a04b4e0327f4656cf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.273218 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0a526f7fced4a04b4e0327f4656cf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.273182 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 71f12a25a1db4650a6c37b5000a668c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.273448 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.274964 8680 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:33.275404 8681 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:33.275490 8679 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:33.275848 8680 tablet_bootstrap.cc:654] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.276023 7217 tablet_service.cc:1467] Processing CreateTablet for tablet a0a526f7fced4a04b4e0327f4656cf69 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.276281 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0a526f7fced4a04b4e0327f4656cf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.276383 8681 tablet_bootstrap.cc:654] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.276414 8679 tablet_bootstrap.cc:654] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.277312 8681 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:33.277407 8681 ts_tablet_manager.cc:1367] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.278013 8681 raft_consensus.cc:348] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.278126 8681 raft_consensus.cc:374] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.278164 8681 raft_consensus.cc:729] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.278271 8681 consensus_queue.cc:260] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.278429 8688 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:33.278544 8681 ts_tablet_manager.cc:1398] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.278652 7220 tablet_service.cc:1467] Processing CreateTablet for tablet 71f12a25a1db4650a6c37b5000a668c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.278883 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.279290 8688 tablet_bootstrap.cc:654] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.279364 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 71f12a25a1db4650a6c37b5000a668c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.279620 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.280428 8688 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:33.280511 8688 ts_tablet_manager.cc:1367] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:33.280915 8688 raft_consensus.cc:348] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.280956 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 527af716906b42729952db28e790bafd (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.281033 8688 raft_consensus.cc:374] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.281076 8688 raft_consensus.cc:729] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.281175 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.281198 8688 consensus_queue.cc:260] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.281493 8688 ts_tablet_manager.cc:1398] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.281664 8688 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:33.281751 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 527af716906b42729952db28e790bafd (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.282040 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.282539 8688 tablet_bootstrap.cc:654] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.283483 8688 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:33.283651 8688 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.284287 8688 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.284443 8688 raft_consensus.cc:374] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.284530 8688 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.284715 8688 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.285009 7483 tablet_service.cc:1467] Processing CreateTablet for tablet 527af716906b42729952db28e790bafd (DEFAULT_TABLE table=rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.285203 8688 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.285256 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.285364 8688 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:33.286087 8688 tablet_bootstrap.cc:654] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.287506 8681 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:33.288336 8688 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:33.288439 8681 tablet_bootstrap.cc:654] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.288470 8688 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:33.288961 8688 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.289172 8688 raft_consensus.cc:374] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.289222 8688 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.289359 8688 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.289534 8681 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:33.289639 8681 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:33.289731 8688 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.290102 8681 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.290210 8681 raft_consensus.cc:374] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.290241 8681 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.290359 8681 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.290627 8681 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.290848 8680 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:33.290941 8680 ts_tablet_manager.cc:1367] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:33.291391 8679 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:33.291385 8680 raft_consensus.cc:348] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.291476 8680 raft_consensus.cc:374] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.291496 8679 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 01:59:33.291510 8680 raft_consensus.cc:729] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.291648 8680 consensus_queue.cc:260] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.291939 8680 ts_tablet_manager.cc:1398] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.292011 8679 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.292112 8680 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:33.292130 8679 raft_consensus.cc:374] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.292171 8679 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.292301 8679 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.292546 8679 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.292721 8679 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:33.292884 8680 tablet_bootstrap.cc:654] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.293582 8679 tablet_bootstrap.cc:654] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.294577 8680 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:33.294657 8680 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:33.295146 8680 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.295229 8680 raft_consensus.cc:374] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.295264 8680 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.295399 8680 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.295711 8680 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.296449 8679 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:33.296550 8679 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:33.296976 8679 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.297071 8679 raft_consensus.cc:374] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.297106 8679 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.297219 8679 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.297464 8679 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.367659 8729 raft_consensus.cc:482] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.367818 8729 raft_consensus.cc:504] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.368265 8729 leader_election.cc:288] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:33.368731 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f12a25a1db4650a6c37b5000a668c7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:33.368728 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f12a25a1db4650a6c37b5000a668c7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:33.368881 7239 raft_consensus.cc:2454] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:33.368896 7634 raft_consensus.cc:2454] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:33.369186 7307 leader_election.cc:302] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.369354 8729 raft_consensus.cc:2781] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.369410 8729 raft_consensus.cc:482] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.369436 8729 raft_consensus.cc:3037] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.370326 8729 raft_consensus.cc:504] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.370667 8729 leader_election.cc:288] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:33.371042 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f12a25a1db4650a6c37b5000a668c7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:33.371057 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f12a25a1db4650a6c37b5000a668c7" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:33.371153 7239 raft_consensus.cc:3037] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.371186 7634 raft_consensus.cc:3037] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.372028 7634 raft_consensus.cc:2454] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:33.372049 7239 raft_consensus.cc:2454] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:33.372332 7307 leader_election.cc:302] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.372512 8729 raft_consensus.cc:2781] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.372606 8729 raft_consensus.cc:686] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:33.372754 8729 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } }
I20240503 01:59:33.374584 7095 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.387215 8396 raft_consensus.cc:482] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.387375 8396 raft_consensus.cc:504] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.387892 8396 leader_election.cc:288] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:33.388437 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0a526f7fced4a04b4e0327f4656cf69" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:33.388437 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0a526f7fced4a04b4e0327f4656cf69" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:33.388604 7370 raft_consensus.cc:2454] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:33.388604 7502 raft_consensus.cc:2454] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:33.388957 7177 leader_election.cc:302] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.389303 8396 raft_consensus.cc:2781] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.389353 8396 raft_consensus.cc:482] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.389393 8396 raft_consensus.cc:3037] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.390374 8396 raft_consensus.cc:504] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.390717 8396 leader_election.cc:288] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:33.391088 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0a526f7fced4a04b4e0327f4656cf69" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:33.391114 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0a526f7fced4a04b4e0327f4656cf69" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:33.391208 7370 raft_consensus.cc:3037] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.391237 7502 raft_consensus.cc:3037] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.392082 7370 raft_consensus.cc:2454] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:33.392128 7502 raft_consensus.cc:2454] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:33.392366 7177 leader_election.cc:302] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.392532 8396 raft_consensus.cc:2781] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.392769 8396 raft_consensus.cc:686] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Running, Role: LEADER
I20240503 01:59:33.392936 8396 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.394969 7095 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.414799 8779 raft_consensus.cc:482] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.414989 8779 raft_consensus.cc:504] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.415478 8779 leader_election.cc:288] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:33.416083 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527af716906b42729952db28e790bafd" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:33.416083 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527af716906b42729952db28e790bafd" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:33.416246 7634 raft_consensus.cc:2454] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:33.416246 7239 raft_consensus.cc:2454] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:33.416565 7438 leader_election.cc:302] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:33.416800 8779 raft_consensus.cc:2781] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.416877 8779 raft_consensus.cc:482] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.416903 8779 raft_consensus.cc:3037] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.417776 8779 raft_consensus.cc:504] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.418154 8779 leader_election.cc:288] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:33.418478 7241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527af716906b42729952db28e790bafd" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:33.418506 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527af716906b42729952db28e790bafd" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:33.418606 7241 raft_consensus.cc:3037] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.418625 7634 raft_consensus.cc:3037] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.419456 7634 raft_consensus.cc:2454] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:33.419468 7241 raft_consensus.cc:2454] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:33.419760 7437 leader_election.cc:302] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:33.419970 8779 raft_consensus.cc:2781] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.420053 8779 raft_consensus.cc:686] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:33.420193 8779 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.422024 7095 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:33.552363 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.552675 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.556226 8797 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 01:59:33.676992 7502 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.677084 7370 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.677666 8794 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.678130 7634 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.678738 8779 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.680485 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.680811 7239 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.680847 7241 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.681068 8794 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.681512 8694 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.682329 8694 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.683071 8779 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 155.387 ms
time per row: 0.0776934 ms
I20240503 01:59:33.763252 7097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47912:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.768740 7350 tablet_service.cc:1467] Processing CreateTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.768869 7483 tablet_service.cc:1467] Processing CreateTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.769110 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.769161 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.769404 7482 tablet_service.cc:1467] Processing CreateTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.769598 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.771324 8680 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:33.771379 8681 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:33.771520 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 66317a254eb644359f65610368d84d76 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.771780 7481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66317a254eb644359f65610368d84d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.771889 7614 tablet_service.cc:1467] Processing CreateTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.772148 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.772259 8681 tablet_bootstrap.cc:654] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.773200 7221 tablet_service.cc:1467] Processing CreateTablet for tablet 66317a254eb644359f65610368d84d76 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.773476 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66317a254eb644359f65610368d84d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.774143 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.774214 8679 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:33.774363 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.774778 8680 tablet_bootstrap.cc:654] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.775125 8679 tablet_bootstrap.cc:654] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.775729 8688 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:33.775887 7352 tablet_service.cc:1467] Processing CreateTablet for tablet 66317a254eb644359f65610368d84d76 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.776115 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66317a254eb644359f65610368d84d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.776551 8688 tablet_bootstrap.cc:654] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.777792 8680 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:33.777876 8680 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:33.778214 8688 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: No bootstrap required, opened a new log
I20240503 01:59:33.778281 8688 ts_tablet_manager.cc:1367] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:33.778321 8680 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.778414 8680 raft_consensus.cc:374] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.778439 8680 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.778551 8680 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.778658 7350 tablet_service.cc:1467] Processing CreateTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.778661 8688 raft_consensus.cc:348] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.778743 8688 raft_consensus.cc:374] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.778770 8688 raft_consensus.cc:729] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.778847 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.778879 8688 consensus_queue.cc:260] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.779143 8688 ts_tablet_manager.cc:1398] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.778851 8680 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.779770 8680 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:33.780756 8680 tablet_bootstrap.cc:654] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.783366 8680 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:33.783453 8680 ts_tablet_manager.cc:1367] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.784363 8679 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:33.784459 8679 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240503 01:59:33.784904 8679 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.785023 8679 raft_consensus.cc:374] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.785063 8679 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.785185 8679 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.785368 8680 raft_consensus.cc:348] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.785470 8680 raft_consensus.cc:374] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.785475 8679 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.785511 8680 raft_consensus.cc:729] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.785665 8679 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:33.785642 8680 consensus_queue.cc:260] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.785897 8680 ts_tablet_manager.cc:1398] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.786067 8680 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 01:59:33.786546 8679 tablet_bootstrap.cc:654] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.786933 8680 tablet_bootstrap.cc:654] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.789801 8679 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58: No bootstrap required, opened a new log
I20240503 01:59:33.789878 8679 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.789815 8680 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: No bootstrap required, opened a new log
I20240503 01:59:33.789969 8680 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.790040 8681 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:33.790127 8681 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240503 01:59:33.790248 8679 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.790324 8679 raft_consensus.cc:374] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.790342 8679 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.790463 8679 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.790601 8681 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.790731 8681 raft_consensus.cc:374] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.790746 8679 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.790757 8681 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.790419 8680 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.790908 8680 raft_consensus.cc:374] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.790894 8681 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.791023 8680 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.791204 8681 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.791185 8680 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.791723 8681 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:33.791882 8680 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:33.792721 8681 tablet_bootstrap.cc:654] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.794023 8681 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:33.794176 8681 ts_tablet_manager.cc:1367] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:33.794759 8681 raft_consensus.cc:348] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.794939 8681 raft_consensus.cc:374] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.795043 8681 raft_consensus.cc:729] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.795262 8681 consensus_queue.cc:260] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.795624 8681 ts_tablet_manager.cc:1398] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.795825 8681 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:33.797013 8681 tablet_bootstrap.cc:654] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.800122 8681 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: No bootstrap required, opened a new log
I20240503 01:59:33.800208 8681 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:33.800568 8681 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.800679 8681 raft_consensus.cc:374] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.800729 8681 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.800853 8681 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:33.801074 8681 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.830123 8794 raft_consensus.cc:482] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.830245 8794 raft_consensus.cc:504] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.830605 8794 leader_election.cc:288] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:33.831043 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66317a254eb644359f65610368d84d76" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:33.831049 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66317a254eb644359f65610368d84d76" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:33.831221 7502 raft_consensus.cc:2454] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:33.831221 7370 raft_consensus.cc:2454] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 0.
I20240503 01:59:33.831508 7177 leader_election.cc:302] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.831735 8794 raft_consensus.cc:2781] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.831792 8794 raft_consensus.cc:482] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.831845 8794 raft_consensus.cc:3037] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.832674 8794 raft_consensus.cc:504] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.832978 8794 leader_election.cc:288] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:33.833346 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66317a254eb644359f65610368d84d76" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:33.833376 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66317a254eb644359f65610368d84d76" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:33.833453 7370 raft_consensus.cc:3037] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.833482 7502 raft_consensus.cc:3037] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.834224 7502 raft_consensus.cc:2454] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:33.834265 7370 raft_consensus.cc:2454] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 1.
I20240503 01:59:33.834481 7177 leader_election.cc:302] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters:
I20240503 01:59:33.834721 8794 raft_consensus.cc:2781] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.834836 8794 raft_consensus.cc:686] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Running, Role: LEADER
I20240503 01:59:33.835003 8794 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } }
I20240503 01:59:33.837091 7098 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.993331 8779 raft_consensus.cc:482] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.993455 8779 raft_consensus.cc:504] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.993887 8779 leader_election.cc:288] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:33.994321 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c539b143674bdd8e3e5aaa4b78c1e9" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:33.994321 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c539b143674bdd8e3e5aaa4b78c1e9" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:33.994489 7370 raft_consensus.cc:2454] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:33.994489 7633 raft_consensus.cc:2454] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 0.
I20240503 01:59:33.994874 7439 leader_election.cc:302] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.995121 8779 raft_consensus.cc:2781] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.995180 8779 raft_consensus.cc:482] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.995234 8779 raft_consensus.cc:3037] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.996335 8779 raft_consensus.cc:504] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:33.996734 8779 leader_election.cc:288] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:33.997073 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c539b143674bdd8e3e5aaa4b78c1e9" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:33.997181 7370 raft_consensus.cc:3037] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.997200 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c539b143674bdd8e3e5aaa4b78c1e9" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:33.997341 7633 raft_consensus.cc:3037] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.997925 7370 raft_consensus.cc:2454] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:33.998092 7633 raft_consensus.cc:2454] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 1.
I20240503 01:59:33.998203 7439 leader_election.cc:302] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:33.998353 8779 raft_consensus.cc:2781] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.998453 8779 raft_consensus.cc:686] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Leader. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:33.998589 8779 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 01:59:34.000514 7098 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:34.073712 8729 raft_consensus.cc:482] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.073841 8729 raft_consensus.cc:504] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:34.074283 8729 leader_election.cc:288] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:34.074658 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b1a4b2ede6493f9e4e094649b4b57d" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:34.074707 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b1a4b2ede6493f9e4e094649b4b57d" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:34.074820 7633 raft_consensus.cc:2454] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:34.074849 7502 raft_consensus.cc:2454] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 0.
I20240503 01:59:34.075125 7306 leader_election.cc:302] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:34.075312 8729 raft_consensus.cc:2781] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.075379 8729 raft_consensus.cc:482] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.075417 8729 raft_consensus.cc:3037] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.076524 8729 raft_consensus.cc:504] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:34.076879 8729 leader_election.cc:288] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:34.077200 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b1a4b2ede6493f9e4e094649b4b57d" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:34.077205 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b1a4b2ede6493f9e4e094649b4b57d" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:34.077315 7502 raft_consensus.cc:3037] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.077327 7633 raft_consensus.cc:3037] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.078155 7502 raft_consensus.cc:2454] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:34.078238 7633 raft_consensus.cc:2454] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 1.
I20240503 01:59:34.078505 7308 leader_election.cc:302] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb, bed387c0ff9241688cdf45f364dc6f5a; no voters:
I20240503 01:59:34.078675 8729 raft_consensus.cc:2781] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.078775 8729 raft_consensus.cc:686] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Leader. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:34.078936 8729 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } }
I20240503 01:59:34.082037 7098 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:34.266604 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.266870 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.270200 8847 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 01:59:34.369257 7370 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.369313 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.369659 8779 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.369944 8664 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.376727 7632 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.377159 8729 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.377249 7501 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.377532 7502 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.378073 7371 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.378659 8794 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.380566 8694 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.381758 8794 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.278 ms
time per row: 0.059639 ms
I20240503 01:59:34.441576 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34145
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:41207
--webserver_port=34155
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 01:59:34.556571 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.556833 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:34.556879 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.559903 8912 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:34.559993 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:34.560062 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:34.563856 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34145
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=34155
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:34.564843 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:34.565989 8912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:34.575362 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:34.576396 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:34.577302 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:34.578263 8912 server_base.cc:1034] running on GCE node
I20240503 01:59:34.578773 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:34.579429 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:34.580590 8912 hybrid_clock.cc:648] HybridClock initialized: now 1714701574580570 us; error 40 us; skew 500 ppm
I20240503 01:59:34.582736 8912 webserver.cc:460] Webserver started at http://127.0.99.133:34155/ using document root <none> and password file <none>
I20240503 01:59:34.583386 8912 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:34.583472 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:34.587347 8912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:34.589650 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.590919 8912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:34.591040 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:34.591624 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:34.610759 8912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:34.611452 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:34.611673 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:34.612349 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:34.613461 8912 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:34.613517 8912 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.613564 8912 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:34.613595 8912 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.622512 8912 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34145
I20240503 01:59:34.622532 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34145 every 8 connection(s)
I20240503 01:59:34.623690 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:34.627848 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 8912
I20240503 01:59:34.628293 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:43615
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:41207
--webserver_port=38471
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:34.633678 9040 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:34.633932 9040 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:34.634554 9040 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:34.636018 7097 ts_manager.cc:194] Re-registered known tserver with Master: e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145)
I20240503 01:59:34.636854 7097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:43325
W20240503 01:59:34.742493 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.742740 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:34.742786 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.746098 9043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:34.746169 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:34.746237 9043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:34.749670 9043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:43615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38471
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:34.750555 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:34.751685 9043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:34.761335 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:34.762365 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:34.763255 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:34.764214 9043 server_base.cc:1034] running on GCE node
I20240503 01:59:34.764698 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:34.765296 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:34.766453 9043 hybrid_clock.cc:648] HybridClock initialized: now 1714701574766431 us; error 35 us; skew 500 ppm
I20240503 01:59:34.768661 9043 webserver.cc:460] Webserver started at http://127.0.99.134:38471/ using document root <none> and password file <none>
I20240503 01:59:34.769254 9043 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:34.769336 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:34.773854 9043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:34.776409 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.777594 9043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:34.777714 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:34.778223 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:34.797394 9043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:34.798027 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:34.798189 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:34.798777 9043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:34.799760 9043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:34.799818 9043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.799881 9043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:34.799917 9043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.808480 9043 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43615
I20240503 01:59:34.808521 9170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43615 every 8 connection(s)
I20240503 01:59:34.809515 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:34.815501 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 9043
I20240503 01:59:34.815989 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:43981
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:41207
--webserver_port=45419
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:34.818275 9171 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:34.818516 9171 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:34.819061 9171 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:34.820113 7097 ts_manager.cc:194] Re-registered known tserver with Master: c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615)
I20240503 01:59:34.820775 7097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:57945
W20240503 01:59:34.927740 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.927978 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:34.928022 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.930950 9174 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:34.931048 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:34.931170 9174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:34.935159 9174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:43981
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=45419
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:34.936164 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:34.937253 9174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:34.948495 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:34.949571 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:34.950585 9174 server_base.cc:1034] running on GCE node
W20240503 01:59:34.951295 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:34.951783 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:34.952452 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:34.953648 9174 hybrid_clock.cc:648] HybridClock initialized: now 1714701574953613 us; error 46 us; skew 500 ppm
I20240503 01:59:34.955775 9174 webserver.cc:460] Webserver started at http://127.0.99.135:45419/ using document root <none> and password file <none>
I20240503 01:59:34.956324 9174 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:34.956391 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:34.959844 9174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 01:59:34.961586 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.962546 9174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:34.962646 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c778431b13784c7c81f3924b7f73e664"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:34.963027 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:34.996006 9174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:34.996716 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:34.996851 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:34.997385 9174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:34.998399 9174 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:34.998450 9174 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:34.998514 9174 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:34.998551 9174 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.006865 9174 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43981
I20240503 01:59:35.006889 9301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43981 every 8 connection(s)
I20240503 01:59:35.007975 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:35.012646 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 9174
I20240503 01:59:35.013087 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8Tfalf/build/debug/bin/kudu
/tmp/dist-test-task8Tfalf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:33955
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:41207
--webserver_port=41281
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 01:59:35.017612 9302 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:35.017882 9302 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.018682 9302 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:35.020079 7097 ts_manager.cc:194] Re-registered known tserver with Master: c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:35.020949 7097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:60709
W20240503 01:59:35.126125 9305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.126351 9305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:35.126417 9305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.129308 9305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:35.129386 9305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:35.129480 9305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:35.133538 9305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:33955
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=41281
--tserver_master_addrs=127.0.99.190:41207
--never_fsync=true
--heap_profile_path=/tmp/kudu.9305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:35.134500 9305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:35.135725 9305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:35.145892 9310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.146641 9311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.147651 9305 server_base.cc:1034] running on GCE node
W20240503 01:59:35.148414 9313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.148803 9305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:35.149350 9305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:35.150490 9305 hybrid_clock.cc:648] HybridClock initialized: now 1714701575150478 us; error 30 us; skew 500 ppm
I20240503 01:59:35.152503 9305 webserver.cc:460] Webserver started at http://127.0.99.136:41281/ using document root <none> and password file <none>
I20240503 01:59:35.153079 9305 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:35.153167 9305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:35.157124 9305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:35.159106 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.160254 9305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:35.160351 9305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "72577fce0313400ea862bf873738af54"
format_stamp: "Formatted at 2024-05-03 01:59:24 on dist-test-slave-pvfz"
I20240503 01:59:35.160832 9305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:35.177955 9305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:35.178658 9305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:35.178812 9305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:35.179348 9305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:35.180250 9305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:35.180296 9305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.180338 9305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:35.180372 9305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.189247 9305 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:33955
I20240503 01:59:35.189292 9432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:33955 every 8 connection(s)
I20240503 01:59:35.190233 9305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:35.199225 9433 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41207
I20240503 01:59:35.199472 9433 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.199461 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8Tfalf/build/debug/bin/kudu as pid 9305
I20240503 01:59:35.200150 9433 heartbeater.cc:511] Master 127.0.99.190:41207 requested a full tablet report, sending...
I20240503 01:59:35.201122 7097 ts_manager.cc:194] Re-registered known tserver with Master: 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:35.201839 7097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:55577
I20240503 01:59:35.557622 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:35.568785 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:35.572654 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:35.576634 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.585649 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.618741 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:35.626462 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.630681 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:35.638978 9040 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:35.653546 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.680696 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:35.685914 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:35.697937 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.701364 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.704103 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:35.712085 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.713228 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:35.716647 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.719612 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.725203 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.730574 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:35.736872 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:35.746927 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.760489 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.761075 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:35.822793 9171 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:35.825640 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.826751 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.832491 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:35.838109 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:35.838675 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:35.857582 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.858060 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.866009 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:35.888300 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:35.894342 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.896385 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.906296 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.909924 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:35.913679 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:35.913851 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:35.916633 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.025461 9302 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:36.046394 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.049494 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.049604 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.062587 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.063510 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.063798 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.098690 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.099644 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.194139 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.203425 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.205029 9433 heartbeater.cc:503] Master 127.0.99.190:41207 was elected leader, sending a full tablet report...
I20240503 01:59:36.210274 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.213290 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.235004 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.240069 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.241786 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.248507 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.253782 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.278873 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.282344 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.290879 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.296214 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.300443 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.307528 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.310458 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.327209 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.327685 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.330891 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.341744 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.344360 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.358009 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.364421 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.369856 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.371960 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.374950 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.375473 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.382032 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.416504 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.417742 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.430460 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.437325 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.494374 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.496978 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.505285 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.507766 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.508409 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.515411 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.525272 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.532500 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.618790 7370 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:36.620563 7501 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.621187 7239 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.621724 10076 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.625670 9522 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.630434 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 01:59:36.637378 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97ffb59d3ab64d03aca5d30a7c3be1e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.639761 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.643414 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.645546 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.668851 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.680094 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.684173 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.684891 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.713074 10076 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:36.718930 7092 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:36.719206 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.719571 7239 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:36.723819 7501 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:36.728849 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.735065 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.741037 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.742089 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.743732 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.749989 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.756326 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.782295 7501 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:36.790534 7370 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.791152 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.791689 8779 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.793244 8779 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.801254 8779 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:36.805953 7633 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
W20240503 01:59:36.809015 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27c539b143674bdd8e3e5aaa4b78c1e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.810428 7094 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:36.813266 7370 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:36.843348 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.851452 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:36.852105 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:36.853675 7239 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.854074 9522 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.855294 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.857549 9522 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:36.858467 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.859015 9522 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.860702 7239 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:36.861829 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.864943 7090 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:36.868923 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:36.875039 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.877281 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.896250 7241 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:36.897943 7632 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.898686 7501 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.898977 8908 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.899389 8908 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.903210 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.903687 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.904428 10203 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
W20240503 01:59:36.905238 7306 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.908814 7502 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:36.911639 7094 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
W20240503 01:59:36.913395 7177 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81086ff737c84d9d9f04743d31c6151b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.920434 7370 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:36.926640 7633 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.927628 10076 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.928471 7241 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.929119 7632 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:36.931151 10077 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.933317 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:36.934487 10077 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:36.935168 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 01:59:36.935451 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad24b47690b4263bdf4a9157db7c17f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.936004 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:36.938995 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:36.941316 7090 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:36.945542 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:36.948289 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.957957 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:36.959627 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.964260 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.989199 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.012182 7239 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.013779 7502 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.014319 7370 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.014739 8908 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.015501 8908 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.017623 8908 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
W20240503 01:59:37.018841 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66317a254eb644359f65610368d84d76. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.019929 7370 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.020637 7090 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.022092 10245 ts_tablet_manager.cc:902] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.022907 10245 tablet_copy_client.cc:308] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.030653 7632 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.033671 7502 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.035679 7370 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.035876 7502 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.036064 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.036773 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.041216 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:37.041364 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-97ffb59d3ab64d03aca5d30a7c3be1e1
I20240503 01:59:37.042831 7392 tablet_copy_source_session.cc:205] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.047482 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.052028 8459 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.052453 7370 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.052798 7502 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.055393 7090 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.055980 10245 tablet_copy_client.cc:783] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.056214 10245 tablet_copy_client.cc:647] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.057904 10245 tablet_copy_client.cc:523] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.060062 10245 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:37.065168 7502 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.067128 7370 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.067847 7632 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.067998 8779 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.068284 10150 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.070152 7632 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.071205 10245 log.cc:824] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:37.071841 7239 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.072357 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.072552 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b15d48b2fa04269b98c6f801f54df4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.073633 7502 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.074018 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.075526 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc68f31e6ef8411e96d46b2419bf9676. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.079470 8779 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:37.081302 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4628afbfeff4f8bbaba5da1e89e4850. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.081914 7633 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.082398 7090 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.085340 7370 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.090513 10270 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.091984 7502 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.095280 7090 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.100277 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.102205 7241 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.102609 9522 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.104079 7632 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.104334 7306 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.104573 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.104828 10235 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.105242 10077 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.106568 7632 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.107334 7241 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.110482 7094 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.114070 7240 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.116394 7370 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.117033 8908 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.117079 7502 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.119609 8908 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.120692 7502 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
W20240503 01:59:37.121111 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18468345c7924d4f88366524b56e8ab3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.122805 7371 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.123894 7632 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.124450 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27c539b143674bdd8e3e5aaa4b78c1e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.124511 8908 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.124644 10150 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.125212 10150 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.126952 10150 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.128002 7633 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.129860 7091 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.130410 7370 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.130712 7090 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.127527 7501 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
W20240503 01:59:37.129276 7436 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27c539b143674bdd8e3e5aaa4b78c1e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.134011 7371 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.137463 7239 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.140722 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.142056 7632 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.144203 7501 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.144546 10295 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.145704 7371 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.146042 10295 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.146507 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 01:59:37.147644 7570 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0539dded7554723b9e918e8cdfbab20. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.149526 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:37.152158 10245 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.152535 10245 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:37.152853 10245 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.093s user 0.033s sys 0.004s
I20240503 01:59:37.155392 10245 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.155694 10245 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:37.156041 10245 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.156970 10245 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:37.158641 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:37.169642 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.170555 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.171936 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-97ffb59d3ab64d03aca5d30a7c3be1e1 received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:37.172029 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-97ffb59d3ab64d03aca5d30a7c3be1e1 on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:37.193948 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.202214 7370 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.203761 7239 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.203893 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:37.205056 7633 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.205504 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.205703 10235 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.205897 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.206871 7634 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.209298 10235 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.211298 7239 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.212056 7502 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.212159 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.212278 7091 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.212469 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.212679 7239 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.214145 8459 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.214476 7502 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.217216 7094 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.217268 7240 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:37.217694 7568 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4628afbfeff4f8bbaba5da1e89e4850. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.219859 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4628afbfeff4f8bbaba5da1e89e4850. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.221451 10295 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.225478 7501 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.225718 7094 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.227257 7371 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.233207 7371 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.234874 7632 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.234930 7634 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.235782 9522 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.237365 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc68f31e6ef8411e96d46b2419bf9676. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.237849 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.238142 7501 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.238269 7371 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.238417 7633 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.238572 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.238799 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc68f31e6ef8411e96d46b2419bf9676. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.238833 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.240981 8459 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.241432 7501 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.244482 7240 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.245507 7090 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.246940 10077 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.248786 7634 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.251536 7094 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.252828 9522 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.253536 7634 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.254551 7240 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.255436 7501 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.255479 7370 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.256407 10295 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.256558 8459 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.256558 7570 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0539dded7554723b9e918e8cdfbab20. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.257313 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0539dded7554723b9e918e8cdfbab20. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.258632 10287 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.259150 7370 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.260785 7501 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.262290 7090 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.262750 7371 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.263207 7634 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.264706 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.264930 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec179e5e683d4ba099c459331ea5e578. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.265056 8459 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.268153 7240 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.268656 10295 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.269637 7370 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.270174 10295 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.271090 7634 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.271750 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.272663 7370 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.273639 7240 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:37.275645 7308 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.275880 7501 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.276377 7091 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.276983 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.277392 10236 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.279455 7370 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
W20240503 01:59:37.282104 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.282449 7240 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.282342 7634 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.282881 10077 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.284662 10077 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
W20240503 01:59:37.285917 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.286096 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.286618 7240 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.286815 7099 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.289191 7096 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.291954 7634 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.292374 10236 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.293283 7634 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.295787 7370 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.297462 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.297787 10235 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.297936 7305 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.299021 7634 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.299294 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.299594 10235 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.300303 7370 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.301067 7371 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.301363 10077 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
W20240503 01:59:37.302438 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d00b8fcb29f47298643903ec0cba606. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.302752 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f35ccb9c21b4c10b9187596d2f6929c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.303475 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.304952 7501 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.305017 7239 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.305346 10235 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.306345 7094 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.306681 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.307206 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.307623 10076 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.307996 7501 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.308176 9522 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.309896 7634 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.311275 7633 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.312497 7239 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.312729 7090 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.316516 7633 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.316516 7370 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.316620 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.318169 7502 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.318217 7501 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.318432 7308 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.318650 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.318691 10295 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.318977 7370 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.319113 7634 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.319252 7307 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.319512 10295 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.319767 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.320123 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be1f4d2622c6452b869f88a301a5e9bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.321238 10236 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.321549 10295 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.321929 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.323607 7370 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.324407 7094 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.324813 7096 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.325423 7634 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.328835 9522 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.330592 7502 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.331275 7239 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.332360 7094 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.333153 7370 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.335635 7633 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.336036 7239 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.336210 7307 consensus_peers.cc:485] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9796308eacd94fb5ad3cb524aa590cfa. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.336367 10236 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.338238 7634 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.338575 10236 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.342378 7239 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.343701 7633 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.343837 7174 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81086ff737c84d9d9f04743d31c6151b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.343842 7502 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.343870 7177 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81086ff737c84d9d9f04743d31c6151b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.344223 8908 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.344719 8908 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.345965 10201 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.347040 7633 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.347891 7502 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.348731 7099 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.350621 10236 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.353466 10461 ts_tablet_manager.cc:902] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.353351 7239 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.354832 10461 tablet_copy_client.cc:308] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.355650 7096 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.357858 7633 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.357961 7502 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.360121 7239 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.360764 7501 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.360850 10339 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.361541 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.361625 7633 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.362172 10340 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.362923 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40260acad62242c3a72197d2dafae187. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.363046 7240 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.363188 7633 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.363605 10339 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.363895 10339 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.367332 10339 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.369064 7239 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.371153 7632 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.371345 7094 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.372243 10151 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.373200 7370 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.373633 7501 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.374155 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95d3c29bfbd4899b27b770bf9d3f4a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.374418 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.374739 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.375048 7632 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.377115 7239 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.379340 7632 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.380070 7096 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.381732 7371 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.381889 10287 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.382428 7501 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.383987 7632 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.384357 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.385823 7308 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68791b3adccd4168b55a2da4423232d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.386016 7090 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.386581 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 8ad24b47690b4263bdf4a9157db7c17f from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:37.386772 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 8ad24b47690b4263bdf4a9157db7c17f from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:37.387245 7501 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.387848 7392 tablet_copy_source_session.cc:205] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.388588 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.390550 7370 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.391484 7502 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.392148 10461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.393661 7370 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.393941 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b15d48b2fa04269b98c6f801f54df4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.394122 10340 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.394325 7634 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.394739 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b15d48b2fa04269b98c6f801f54df4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.394908 10340 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.395792 10340 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.397074 7634 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.397802 7370 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.400014 7371 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.400090 7096 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
W20240503 01:59:37.402562 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f35ccb9c21b4c10b9187596d2f6929c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.403036 7634 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.403406 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.403643 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f35ccb9c21b4c10b9187596d2f6929c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.403941 7239 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.404342 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.405305 10236 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.406369 7239 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.407469 7634 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.407635 10461 tablet_copy_client.cc:783] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.407895 10461 tablet_copy_client.cc:647] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.410802 7090 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.411409 10461 tablet_copy_client.cc:523] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.412575 10442 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.413390 10461 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.414778 7501 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.417330 7094 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.420761 10461 log.cc:824] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:37.421900 7632 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.427358 7239 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.430171 7371 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.430575 7501 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.430567 10203 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.430992 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0a526f7fced4a04b4e0327f4656cf69. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.431174 10203 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.432907 10203 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.435712 7099 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.435920 7371 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.439851 7501 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.448616 7370 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.448616 7371 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.450270 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.450501 7501 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:37.450523 7632 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.450739 7307 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad24b47690b4263bdf4a9157db7c17f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.450902 10077 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.451138 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:37.451274 7307 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.451488 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.451766 10236 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.452075 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.452239 7632 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:37.452649 10076 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.452649 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20240503 01:59:37.455284 7308 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.455343 10245 ts_tablet_manager.cc:902] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:37.455837 10245 tablet_copy_client.cc:308] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 01:59:37.455789 9522 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.456683 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.458094 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.458585 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.459393 7634 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.460676 7094 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.466090 7096 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.471042 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 66317a254eb644359f65610368d84d76 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:32955)
I20240503 01:59:37.471200 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 66317a254eb644359f65610368d84d76 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:32955: session id = c778431b13784c7c81f3924b7f73e664-66317a254eb644359f65610368d84d76
I20240503 01:59:37.472602 7261 tablet_copy_source_session.cc:205] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.473870 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66317a254eb644359f65610368d84d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.476061 10245 tablet_copy_client.cc:783] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.476217 10245 tablet_copy_client.cc:647] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.477871 10245 tablet_copy_client.cc:523] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.479219 10245 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:37.480283 7371 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.481925 7239 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.482437 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d332a7c255b44589d54b1f583a2f9c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.482745 7501 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.482777 10442 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.484158 10442 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.484828 10236 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.488530 7239 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.491000 7094 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.495429 7502 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.497279 7239 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.497287 7436 consensus_peers.cc:485] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb00fcf563b74243b7c955ddacf1d6cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.497785 7371 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.497977 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.498618 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.500253 7502 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.502892 7502 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
W20240503 01:59:37.505162 7436 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:37.506137 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.506457 10461 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.506973 7634 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.507562 10461 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:37.507813 7240 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.508246 8779 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.509513 10150 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.509620 10335 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.510267 7239 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.512863 7371 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.514173 7090 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.514750 10150 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.515949 7634 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.518391 7239 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.518571 7094 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:37.511693 10461 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.098s user 0.035s sys 0.003s
I20240503 01:59:37.523715 10461 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.524066 10461 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:37.524502 10461 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.532832 10461 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.013s user 0.002s sys 0.002s
I20240503 01:59:37.533886 10617 ts_tablet_manager.cc:902] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:37.534642 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-8ad24b47690b4263bdf4a9157db7c17f received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:37.534740 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-8ad24b47690b4263bdf4a9157db7c17f on tablet 8ad24b47690b4263bdf4a9157db7c17f with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:37.535421 10617 tablet_copy_client.cc:308] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:37.542891 10461 ts_tablet_manager.cc:902] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:37.543339 10461 tablet_copy_client.cc:308] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 01:59:37.551275 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 18468345c7924d4f88366524b56e8ab3 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:57905)
I20240503 01:59:37.551398 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 18468345c7924d4f88366524b56e8ab3 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:57905: session id = e648eca559ed4e5ab24a1b98755aa3eb-18468345c7924d4f88366524b56e8ab3
I20240503 01:59:37.552469 7261 tablet_copy_source_session.cc:205] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.553571 10461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18468345c7924d4f88366524b56e8ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.554006 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:60945)
I20240503 01:59:37.554138 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:60945: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-27c539b143674bdd8e3e5aaa4b78c1e9
I20240503 01:59:37.555791 7523 tablet_copy_source_session.cc:205] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.556051 10461 tablet_copy_client.cc:783] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.556188 10461 tablet_copy_client.cc:647] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.556406 10245 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.556758 10245 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:37.557021 10245 ts_tablet_manager.cc:1367] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.078s user 0.028s sys 0.000s
I20240503 01:59:37.557723 10245 raft_consensus.cc:348] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.557919 10245 raft_consensus.cc:729] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:37.558070 10245 consensus_queue.cc:260] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.558614 10245 ts_tablet_manager.cc:1398] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.559185 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-66317a254eb644359f65610368d84d76 received from {username='slave'} at 127.0.99.135:32955
I20240503 01:59:37.559306 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session c778431b13784c7c81f3924b7f73e664-66317a254eb644359f65610368d84d76 on tablet 66317a254eb644359f65610368d84d76 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:37.559427 10461 tablet_copy_client.cc:523] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.560555 10617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.560880 10461 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.565523 7502 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:37.567392 7632 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.568307 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ebe541161c84bd6b11bcd3d13d3bd2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.568971 8779 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.569099 7634 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
W20240503 01:59:37.570852 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95d3c29bfbd4899b27b770bf9d3f4a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.571017 7371 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.571694 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95d3c29bfbd4899b27b770bf9d3f4a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.571710 7502 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.571928 10295 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.572096 10270 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.572844 7239 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.574312 10339 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.575996 7371 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.578334 10295 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.578866 7092 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.583248 10647 ts_tablet_manager.cc:902] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.584229 10647 tablet_copy_client.cc:308] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.585444 9256 raft_consensus.cc:1203] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:37.587805 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 71f12a25a1db4650a6c37b5000a668c7 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:37.587903 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 71f12a25a1db4650a6c37b5000a668c7 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-71f12a25a1db4650a6c37b5000a668c7
I20240503 01:59:37.588826 7392 tablet_copy_source_session.cc:205] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.592818 8779 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.595803 7240 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.598456 7099 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.600765 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.600812 7632 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:37.605978 10647 tablet_copy_client.cc:783] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.608428 10647 tablet_copy_client.cc:647] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.610842 10647 tablet_copy_client.cc:523] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.612373 10647 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.615034 7370 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
W20240503 01:59:37.617364 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d332a7c255b44589d54b1f583a2f9c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.618184 7501 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.618666 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d332a7c255b44589d54b1f583a2f9c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.618822 10442 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.620182 7240 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.620393 10442 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.620536 10077 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.620820 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:37.622965 7501 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.625795 7096 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.626734 7240 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.630867 7502 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.647059 10617 tablet_copy_client.cc:783] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.647368 10617 tablet_copy_client.cc:647] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.650044 10617 tablet_copy_client.cc:523] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.652781 10617 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:37.663569 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.679888 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.683540 10647 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.684391 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:37.685711 10647 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:37.686075 10647 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.010s
I20240503 01:59:37.686738 10647 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.691062 10647 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:37.691301 10647 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.692454 10699 ts_tablet_manager.cc:902] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:37.692839 10647 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 01:59:37.693709 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-71f12a25a1db4650a6c37b5000a668c7 received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:37.693789 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-71f12a25a1db4650a6c37b5000a668c7 on tablet 71f12a25a1db4650a6c37b5000a668c7 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:37.694473 10699 tablet_copy_client.cc:308] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:37.696933 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.697947 10617 log.cc:824] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:37.698805 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:37.698907 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-27c539b143674bdd8e3e5aaa4b78c1e9
I20240503 01:59:37.699810 7523 tablet_copy_source_session.cc:205] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.700712 10699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.702158 10461 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.702435 10461 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:37.702600 10461 ts_tablet_manager.cc:1367] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.142s user 0.018s sys 0.006s
I20240503 01:59:37.702821 10699 tablet_copy_client.cc:783] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.703018 10699 tablet_copy_client.cc:647] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.704107 10699 tablet_copy_client.cc:523] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.705338 10699 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.707762 10461 raft_consensus.cc:348] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.707955 10461 raft_consensus.cc:729] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:37.708124 10461 consensus_queue.cc:260] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.715899 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:37.712409 10461 ts_tablet_manager.cc:1398] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20240503 01:59:37.717207 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-18468345c7924d4f88366524b56e8ab3 received from {username='slave'} at 127.0.99.133:57905
I20240503 01:59:37.717304 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-18468345c7924d4f88366524b56e8ab3 on tablet 18468345c7924d4f88366524b56e8ab3 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:37.720713 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.721128 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.722610 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.723598 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:37.725560 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:37.726517 7632 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:37.726734 10076 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:37.727444 10076 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:37.727896 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.728063 10720 ts_tablet_manager.cc:902] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.728646 10720 tablet_copy_client.cc:308] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.729840 10076 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.730618 7632 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.732265 10670 ts_tablet_manager.cc:902] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.733673 7092 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.734557 10670 tablet_copy_client.cc:308] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
W20240503 01:59:37.734659 7305 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.735014 10673 ts_tablet_manager.cc:902] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.735595 10673 tablet_copy_client.cc:308] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.735718 10672 ts_tablet_manager.cc:902] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:37.748741 7632 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.749264 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:37.750993 7501 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:37.751050 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be1f4d2622c6452b869f88a301a5e9bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.751514 10287 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:37.752526 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be1f4d2622c6452b869f88a301a5e9bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.752784 7370 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:37.753175 10287 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:37.754123 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 71f12a25a1db4650a6c37b5000a668c7 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:37.754223 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 71f12a25a1db4650a6c37b5000a668c7 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-71f12a25a1db4650a6c37b5000a668c7
I20240503 01:59:37.755131 7392 tablet_copy_source_session.cc:205] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.755460 10245 ts_tablet_manager.cc:902] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.755987 10245 tablet_copy_client.cc:308] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
W20240503 01:59:37.757241 7308 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.757192 10287 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.757568 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:45331)
I20240503 01:59:37.757717 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:45331: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-b4628afbfeff4f8bbaba5da1e89e4850
I20240503 01:59:37.759248 7653 tablet_copy_source_session.cc:205] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.757570 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet cc68f31e6ef8411e96d46b2419bf9676 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:45331)
I20240503 01:59:37.760259 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet cc68f31e6ef8411e96d46b2419bf9676 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:45331: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-cc68f31e6ef8411e96d46b2419bf9676
I20240503 01:59:37.760738 7092 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:37.760857 7370 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.761030 7654 tablet_copy_source_session.cc:205] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.762545 10720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc68f31e6ef8411e96d46b2419bf9676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.765143 10720 tablet_copy_client.cc:783] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.765305 10720 tablet_copy_client.cc:647] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.765864 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:37.768292 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 9796308eacd94fb5ad3cb524aa590cfa from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:37.768405 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 9796308eacd94fb5ad3cb524aa590cfa from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-9796308eacd94fb5ad3cb524aa590cfa
I20240503 01:59:37.769230 7392 tablet_copy_source_session.cc:205] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.770277 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9796308eacd94fb5ad3cb524aa590cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.772552 10245 tablet_copy_client.cc:783] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.772709 10245 tablet_copy_client.cc:647] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.774277 7502 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.778203 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.778403 10720 tablet_copy_client.cc:523] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.779995 10720 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:37.782294 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.787140 10673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.787232 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.789196 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.789675 10673 tablet_copy_client.cc:783] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.789916 10673 tablet_copy_client.cc:647] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.790984 10673 tablet_copy_client.cc:523] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.792152 10672 tablet_copy_client.cc:308] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 01:59:37.793208 10461 ts_tablet_manager.cc:902] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.793793 10461 tablet_copy_client.cc:308] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.794275 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 0d00b8fcb29f47298643903ec0cba606 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:37.794443 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 0d00b8fcb29f47298643903ec0cba606 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-0d00b8fcb29f47298643903ec0cba606
I20240503 01:59:37.795821 7392 tablet_copy_source_session.cc:205] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.796977 10461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d00b8fcb29f47298643903ec0cba606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.799222 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 81086ff737c84d9d9f04743d31c6151b from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:56063)
I20240503 01:59:37.799316 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 81086ff737c84d9d9f04743d31c6151b from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:56063: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-81086ff737c84d9d9f04743d31c6151b
I20240503 01:59:37.799417 10461 tablet_copy_client.cc:783] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.799616 10461 tablet_copy_client.cc:647] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.800177 7261 tablet_copy_source_session.cc:205] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.802325 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:37.803507 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:37.814280 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.822086 10670 tablet_copy_client.cc:783] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.822516 10670 tablet_copy_client.cc:647] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.823115 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:37.825845 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.843828 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:37.847342 10670 tablet_copy_client.cc:523] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.849092 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.849872 10670 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:37.850637 10768 ts_tablet_manager.cc:902] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.851235 10768 tablet_copy_client.cc:308] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.860499 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet be1f4d2622c6452b869f88a301a5e9bc from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:59397)
I20240503 01:59:37.860630 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet be1f4d2622c6452b869f88a301a5e9bc from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:59397: session id = 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:37.861595 7653 tablet_copy_source_session.cc:205] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.862900 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.864193 9367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.867805 10673 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:37.867996 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:37.870124 10647 ts_tablet_manager.cc:902] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:37.870647 10647 tablet_copy_client.cc:308] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:37.871199 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 3b15d48b2fa04269b98c6f801f54df4f from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:37.871310 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 3b15d48b2fa04269b98c6f801f54df4f from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-3b15d48b2fa04269b98c6f801f54df4f
I20240503 01:59:37.872287 7523 tablet_copy_source_session.cc:205] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.873227 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b15d48b2fa04269b98c6f801f54df4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.876763 10767 ts_tablet_manager.cc:902] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.875543 10647 tablet_copy_client.cc:783] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.876947 10647 tablet_copy_client.cc:647] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.877758 10759 ts_tablet_manager.cc:902] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.878360 10762 ts_tablet_manager.cc:902] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.882491 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:37.883481 8994 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:37.891532 10775 ts_tablet_manager.cc:902] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.892397 10776 ts_tablet_manager.cc:902] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.900243 10672 tablet_copy_client.cc:783] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.900228 10670 log.cc:824] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:37.900602 10672 tablet_copy_client.cc:647] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.901947 10768 tablet_copy_client.cc:783] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.902037 10672 tablet_copy_client.cc:523] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.902220 10768 tablet_copy_client.cc:647] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.907804 10672 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:37.909117 10797 ts_tablet_manager.cc:902] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.909291 10768 tablet_copy_client.cc:523] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.911085 10461 tablet_copy_client.cc:523] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.911223 10797 tablet_copy_client.cc:308] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.911803 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:37.911932 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-5f35ccb9c21b4c10b9187596d2f6929c
I20240503 01:59:37.912667 10461 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.912894 7392 tablet_copy_source_session.cc:205] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.914000 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.915318 10775 tablet_copy_client.cc:308] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.916468 10797 tablet_copy_client.cc:783] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.916631 10797 tablet_copy_client.cc:647] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.917238 10768 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:37.918023 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet f0539dded7554723b9e918e8cdfbab20 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:45331)
I20240503 01:59:37.918138 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet f0539dded7554723b9e918e8cdfbab20 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:45331: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-f0539dded7554723b9e918e8cdfbab20
I20240503 01:59:37.918618 10808 ts_tablet_manager.cc:902] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:37.919039 7653 tablet_copy_source_session.cc:205] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.919399 10808 tablet_copy_client.cc:308] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 01:59:37.919718 10777 ts_tablet_manager.cc:902] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.920049 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 81086ff737c84d9d9f04743d31c6151b from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:57905)
I20240503 01:59:37.920167 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 81086ff737c84d9d9f04743d31c6151b from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:57905: session id = e648eca559ed4e5ab24a1b98755aa3eb-81086ff737c84d9d9f04743d31c6151b
I20240503 01:59:37.920225 10777 tablet_copy_client.cc:308] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.921108 7261 tablet_copy_source_session.cc:205] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.923161 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:59397)
I20240503 01:59:37.923261 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:59397: session id = 72577fce0313400ea862bf873738af54-b4628afbfeff4f8bbaba5da1e89e4850
I20240503 01:59:37.923790 10797 tablet_copy_client.cc:523] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.923843 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:37.924131 7653 tablet_copy_source_session.cc:205] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.925190 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.925567 10797 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:37.927615 10777 tablet_copy_client.cc:783] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.927757 10777 tablet_copy_client.cc:647] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.929258 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.931645 10775 tablet_copy_client.cc:783] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.931833 10775 tablet_copy_client.cc:647] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.954277 10768 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.954810 10768 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:37.955302 10768 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.000s
I20240503 01:59:37.958489 10768 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.958959 10768 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:37.959362 10768 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:37.960538 10768 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:37.962531 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc received from {username='slave'} at 127.0.99.136:59397
I20240503 01:59:37.962689 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc on tablet be1f4d2622c6452b869f88a301a5e9bc with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:37.964618 10768 ts_tablet_manager.cc:902] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:37.964862 10798 ts_tablet_manager.cc:902] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:37.965210 10768 tablet_copy_client.cc:308] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.965350 10798 tablet_copy_client.cc:308] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.965866 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet ec179e5e683d4ba099c459331ea5e578 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:59397)
I20240503 01:59:37.965963 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet ec179e5e683d4ba099c459331ea5e578 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:59397: session id = 72577fce0313400ea862bf873738af54-ec179e5e683d4ba099c459331ea5e578
I20240503 01:59:37.966929 7653 tablet_copy_source_session.cc:205] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.967860 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.969812 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:37.969931 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:37.970551 10768 tablet_copy_client.cc:783] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.970760 10768 tablet_copy_client.cc:647] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.970922 7392 tablet_copy_source_session.cc:205] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.970942 10245 tablet_copy_client.cc:523] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.971946 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.972278 10768 tablet_copy_client.cc:523] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.972474 10245 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:37.973011 10617 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.973380 10617 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:37.973551 10768 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:37.973709 10617 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.321s user 0.031s sys 0.005s
I20240503 01:59:37.974812 10808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.976452 10617 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:37.977479 9256 raft_consensus.cc:1203] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:37.980460 10808 tablet_copy_client.cc:783] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.981135 10767 tablet_copy_client.cc:308] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.981189 10762 tablet_copy_client.cc:308] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:37.981135 10759 tablet_copy_client.cc:308] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.981346 10776 tablet_copy_client.cc:308] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:37.981920 10808 tablet_copy_client.cc:647] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.981992 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a432cd5fe20941d6bc73dae386d85170 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:37.982025 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:37.982120 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-5f35ccb9c21b4c10b9187596d2f6929c
I20240503 01:59:37.982808 10824 ts_tablet_manager.cc:902] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:37.983240 7391 tablet_copy_source_session.cc:205] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.983579 10824 tablet_copy_client.cc:308] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:37.983942 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a432cd5fe20941d6bc73dae386d85170 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:37.984329 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.985002 7392 tablet_copy_source_session.cc:205] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.986101 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.986672 10759 tablet_copy_client.cc:783] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.986825 10759 tablet_copy_client.cc:647] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.988041 10759 tablet_copy_client.cc:523] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.988373 10776 tablet_copy_client.cc:783] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.988519 10776 tablet_copy_client.cc:647] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.989362 10759 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:37.990906 10776 tablet_copy_client.cc:523] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.991757 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 40260acad62242c3a72197d2dafae187 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:50205)
I20240503 01:59:37.991897 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 40260acad62242c3a72197d2dafae187 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:50205: session id = 72577fce0313400ea862bf873738af54-40260acad62242c3a72197d2dafae187
I20240503 01:59:37.992455 10776 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:37.992805 7523 tablet_copy_source_session.cc:205] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.994793 10824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.005089 10777 tablet_copy_client.cc:523] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.006569 10617 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.006677 10777 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.009058 10798 tablet_copy_client.cc:783] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.009315 10798 tablet_copy_client.cc:647] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.011831 10798 tablet_copy_client.cc:523] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.013566 10798 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.004297 10837 ts_tablet_manager.cc:902] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:38.016018 10837 tablet_copy_client.cc:308] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:37.982128 10775 tablet_copy_client.cc:523] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.016606 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet fb00fcf563b74243b7c955ddacf1d6cf from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:38.016736 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet fb00fcf563b74243b7c955ddacf1d6cf from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-fb00fcf563b74243b7c955ddacf1d6cf
I20240503 01:59:38.017740 7523 tablet_copy_source_session.cc:205] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.020021 10775 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.020426 10768 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.020747 10768 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.020958 10768 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.047s user 0.028s sys 0.000s
I20240503 01:59:38.020987 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb00fcf563b74243b7c955ddacf1d6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.021593 10768 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.021719 10768 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.021829 10768 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.022825 10768 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:38.023638 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.024451 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 72577fce0313400ea862bf873738af54-ec179e5e683d4ba099c459331ea5e578 received from {username='slave'} at 127.0.99.136:59397
I20240503 01:59:38.024554 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 72577fce0313400ea862bf873738af54-ec179e5e683d4ba099c459331ea5e578 on tablet ec179e5e683d4ba099c459331ea5e578 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.026791 10837 tablet_copy_client.cc:783] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.026958 10837 tablet_copy_client.cc:647] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.027017 10808 tablet_copy_client.cc:523] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.028131 10768 ts_tablet_manager.cc:902] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:38.028182 10808 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:38.028604 10768 tablet_copy_client.cc:308] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:38.029064 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 3b15d48b2fa04269b98c6f801f54df4f from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:50205)
I20240503 01:59:38.029186 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 3b15d48b2fa04269b98c6f801f54df4f from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:50205: session id = 72577fce0313400ea862bf873738af54-3b15d48b2fa04269b98c6f801f54df4f
I20240503 01:59:38.030135 7523 tablet_copy_source_session.cc:205] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.030205 10824 tablet_copy_client.cc:783] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.030354 10824 tablet_copy_client.cc:647] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.031373 10719 ts_tablet_manager.cc:902] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.033963 10837 tablet_copy_client.cc:523] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.035128 8994 raft_consensus.cc:1203] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.036047 10617 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.037175 10617 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20240503 01:59:38.037993 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-27c539b143674bdd8e3e5aaa4b78c1e9 received from {username='slave'} at 127.0.99.134:60945
I20240503 01:59:38.038103 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-27c539b143674bdd8e3e5aaa4b78c1e9 on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.040939 10837 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:38.047294 10760 ts_tablet_manager.cc:902] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.047919 10760 tablet_copy_client.cc:308] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.048179 10769 ts_tablet_manager.cc:902] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.048519 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 68791b3adccd4168b55a2da4423232d7 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:38.048645 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 68791b3adccd4168b55a2da4423232d7 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-68791b3adccd4168b55a2da4423232d7
I20240503 01:59:38.048851 10769 tablet_copy_client.cc:308] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.049326 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a432cd5fe20941d6bc73dae386d85170 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:38.049418 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a432cd5fe20941d6bc73dae386d85170 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:38.049540 7392 tablet_copy_source_session.cc:205] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.050263 7391 tablet_copy_source_session.cc:205] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.051194 10769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.054168 10774 ts_tablet_manager.cc:902] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 01:59:38.054270 10673 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.054600 10673 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.054770 10673 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.187s user 0.024s sys 0.001s
I20240503 01:59:38.055300 10774 tablet_copy_client.cc:308] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 01:59:38.055946 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet a0a526f7fced4a04b4e0327f4656cf69 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:56063)
I20240503 01:59:38.056051 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet a0a526f7fced4a04b4e0327f4656cf69 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:56063: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0a526f7fced4a04b4e0327f4656cf69
I20240503 01:59:38.056986 7261 tablet_copy_source_session.cc:205] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.055334 10673 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.057874 10673 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.057987 10673 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.058286 10673 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:38.058387 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.059125 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-b4628afbfeff4f8bbaba5da1e89e4850 received from {username='slave'} at 127.0.99.134:45331
I20240503 01:59:38.059216 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-b4628afbfeff4f8bbaba5da1e89e4850 on tablet b4628afbfeff4f8bbaba5da1e89e4850 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.060997 10774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0a526f7fced4a04b4e0327f4656cf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.063915 10874 ts_tablet_manager.cc:902] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.031826 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b15d48b2fa04269b98c6f801f54df4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.064611 10874 tablet_copy_client.cc:308] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.065222 10719 tablet_copy_client.cc:308] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.065302 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:38.065410 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:38.065760 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:38.065882 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:38.066418 7391 tablet_copy_source_session.cc:205] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.066751 7392 tablet_copy_source_session.cc:205] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.067009 10768 tablet_copy_client.cc:783] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.067178 10768 tablet_copy_client.cc:647] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.032078 10824 tablet_copy_client.cc:523] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.067306 10879 ts_tablet_manager.cc:902] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.067991 10879 tablet_copy_client.cc:308] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.068635 10824 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.068909 8994 raft_consensus.cc:1203] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.047147 10729 ts_tablet_manager.cc:902] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.047897 10670 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.073670 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 8ad24b47690b4263bdf4a9157db7c17f from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:38.073721 10670 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.073782 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 8ad24b47690b4263bdf4a9157db7c17f from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:38.073902 10670 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.224s user 0.030s sys 0.004s
I20240503 01:59:38.074401 10670 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.074530 10670 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.074632 10670 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.074746 7392 tablet_copy_source_session.cc:205] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.074893 10670 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.076557 10729 tablet_copy_client.cc:308] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.077666 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-71f12a25a1db4650a6c37b5000a668c7 received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:38.077711 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 7d332a7c255b44589d54b1f583a2f9c4 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:38.077751 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-71f12a25a1db4650a6c37b5000a668c7 on tablet 71f12a25a1db4650a6c37b5000a668c7 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.078142 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 7d332a7c255b44589d54b1f583a2f9c4 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-7d332a7c255b44589d54b1f583a2f9c4
I20240503 01:59:38.079001 7391 tablet_copy_source_session.cc:205] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.079676 9124 raft_consensus.cc:1203] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.080937 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.081949 10672 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.082517 10647 tablet_copy_client.cc:523] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.084022 10647 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:38.087755 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d332a7c255b44589d54b1f583a2f9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.090250 10729 tablet_copy_client.cc:783] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.090487 10729 tablet_copy_client.cc:647] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.092049 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.094521 10719 tablet_copy_client.cc:783] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.099216 10797 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.083524 10879 tablet_copy_client.cc:783] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.099776 10879 tablet_copy_client.cc:647] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.083859 10874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.088730 10760 tablet_copy_client.cc:783] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.101401 10760 tablet_copy_client.cc:647] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.101770 10672 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.102502 10874 tablet_copy_client.cc:783] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.102689 10874 tablet_copy_client.cc:647] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.103884 10672 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.196s user 0.025s sys 0.003s
I20240503 01:59:38.104151 10899 ts_tablet_manager.cc:902] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:38.102658 10760 tablet_copy_client.cc:523] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.105726 10760 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.097160 10798 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.109787 10798 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.110057 10798 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.097s user 0.027s sys 0.004s
I20240503 01:59:38.110546 10798 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.110810 10798 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.110965 10798 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.111483 10798 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.111745 10879 tablet_copy_client.cc:523] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.112607 10672 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.112743 10672 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.112845 10672 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.113459 10879 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.114081 10899 tablet_copy_client.cc:308] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:38.114629 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 527af716906b42729952db28e790bafd from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:38.114749 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 527af716906b42729952db28e790bafd from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-527af716906b42729952db28e790bafd
I20240503 01:59:38.114807 10672 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.115509 7523 tablet_copy_source_session.cc:205] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.116477 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.116823 10874 tablet_copy_client.cc:523] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.117426 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0b1a4b2ede6493f9e4e094649b4b57d received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:38.117516 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0b1a4b2ede6493f9e4e094649b4b57d on tablet a0b1a4b2ede6493f9e4e094649b4b57d with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.118147 10874 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.126197 10461 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.126608 10461 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.126853 10461 ts_tablet_manager.cc:1367] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.214s user 0.020s sys 0.008s
I20240503 01:59:38.127319 10461 raft_consensus.cc:348] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.127538 10461 raft_consensus.cc:729] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.127687 10461 consensus_queue.cc:260] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.128322 10461 ts_tablet_manager.cc:1398] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:38.129700 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-81086ff737c84d9d9f04743d31c6151b received from {username='slave'} at 127.0.99.134:56063
I20240503 01:59:38.129791 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-81086ff737c84d9d9f04743d31c6151b on tablet 81086ff737c84d9d9f04743d31c6151b with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.139492 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet f0539dded7554723b9e918e8cdfbab20 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:39201)
I20240503 01:59:38.139660 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet f0539dded7554723b9e918e8cdfbab20 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:39201: session id = c778431b13784c7c81f3924b7f73e664-f0539dded7554723b9e918e8cdfbab20
I20240503 01:59:38.140558 7653 tablet_copy_source_session.cc:205] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.140888 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet cc68f31e6ef8411e96d46b2419bf9676 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:39201)
I20240503 01:59:38.140978 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet cc68f31e6ef8411e96d46b2419bf9676 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:39201: session id = c778431b13784c7c81f3924b7f73e664-cc68f31e6ef8411e96d46b2419bf9676
I20240503 01:59:38.141832 7654 tablet_copy_source_session.cc:205] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.143151 10767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.145514 10767 tablet_copy_client.cc:783] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.145670 10767 tablet_copy_client.cc:647] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.146036 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc68f31e6ef8411e96d46b2419bf9676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.146816 10767 tablet_copy_client.cc:523] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.147708 10797 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.147935 10797 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.223s user 0.021s sys 0.007s
I20240503 01:59:38.148221 10767 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.148522 10797 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.148663 10797 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.148772 10797 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.149094 10762 tablet_copy_client.cc:783] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.149174 10797 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.149233 10762 tablet_copy_client.cc:647] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.150851 10762 tablet_copy_client.cc:523] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.152256 10762 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.153159 10899 tablet_copy_client.cc:783] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.153314 10899 tablet_copy_client.cc:647] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.154340 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-0d00b8fcb29f47298643903ec0cba606 received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:38.154425 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-0d00b8fcb29f47298643903ec0cba606 on tablet 0d00b8fcb29f47298643903ec0cba606 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.154934 10899 tablet_copy_client.cc:523] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.156605 10899 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:38.157191 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-5f35ccb9c21b4c10b9187596d2f6929c received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:38.157264 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-5f35ccb9c21b4c10b9187596d2f6929c on tablet 5f35ccb9c21b4c10b9187596d2f6929c with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.160938 10235 raft_consensus.cc:1050] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:38.161458 10235 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:38.163180 7240 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.163226 9256 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.163595 7502 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.163663 10077 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.164063 10077 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.166631 10077 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.171811 10647 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.178308 10077 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.181114 7502 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.181908 7093 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.183084 10647 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.183284 10647 ts_tablet_manager.cc:1367] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.006s
I20240503 01:59:38.184072 10647 raft_consensus.cc:348] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.184228 10647 raft_consensus.cc:729] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.183912 9256 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.184321 10647 consensus_queue.cc:260] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.184358 10769 tablet_copy_client.cc:783] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.184537 10769 tablet_copy_client.cc:647] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.100476 10670 ts_tablet_manager.cc:902] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:38.185381 10670 tablet_copy_client.cc:308] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:38.185484 10776 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.101398 10729 tablet_copy_client.cc:523] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.185715 10776 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.185856 10776 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.193s user 0.022s sys 0.003s
I20240503 01:59:38.186259 10776 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.186374 10776 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.186475 10776 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.186765 10776 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.186880 10647 ts_tablet_manager.cc:1398] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:38.187064 10729 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.187579 10769 tablet_copy_client.cc:523] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.187888 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-3b15d48b2fa04269b98c6f801f54df4f received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:38.187958 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-3b15d48b2fa04269b98c6f801f54df4f on tablet 3b15d48b2fa04269b98c6f801f54df4f with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.188726 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-a432cd5fe20941d6bc73dae386d85170 received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:38.188802 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-a432cd5fe20941d6bc73dae386d85170 on tablet a432cd5fe20941d6bc73dae386d85170 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.188985 10769 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.119612 10824 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.124491 9387 raft_consensus.cc:1203] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.190485 10824 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.190649 10824 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.122s user 0.026s sys 0.000s
I20240503 01:59:38.191063 10824 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.191169 10824 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.191251 10824 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.191780 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 527af716906b42729952db28e790bafd from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:50205)
I20240503 01:59:38.191895 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 527af716906b42729952db28e790bafd from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:50205: session id = 72577fce0313400ea862bf873738af54-527af716906b42729952db28e790bafd
I20240503 01:59:38.191506 10824 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.169452 10768 tablet_copy_client.cc:523] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.192741 7523 tablet_copy_source_session.cc:205] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.193537 10768 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.195988 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session 72577fce0313400ea862bf873738af54-40260acad62242c3a72197d2dafae187 received from {username='slave'} at 127.0.99.136:50205
I20240503 01:59:38.196054 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session 72577fce0313400ea862bf873738af54-40260acad62242c3a72197d2dafae187 on tablet 40260acad62242c3a72197d2dafae187 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.196740 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.198752 10670 tablet_copy_client.cc:783] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.198877 10670 tablet_copy_client.cc:647] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.199669 10824 ts_tablet_manager.cc:902] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:38.200033 10824 tablet_copy_client.cc:308] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:38.201808 10719 tablet_copy_client.cc:647] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.202459 10672 ts_tablet_manager.cc:902] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:38.202877 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:50205)
I20240503 01:59:38.202980 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:50205: session id = 72577fce0313400ea862bf873738af54-8ebe541161c84bd6b11bcd3d13d3bd2e
I20240503 01:59:38.203949 7523 tablet_copy_source_session.cc:205] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.204620 10798 ts_tablet_manager.cc:902] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.204852 10824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.207238 10824 tablet_copy_client.cc:783] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.207434 10824 tablet_copy_client.cc:647] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.207691 10719 tablet_copy_client.cc:523] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.208174 10798 tablet_copy_client.cc:308] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.208655 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 7d332a7c255b44589d54b1f583a2f9c4 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:38.208784 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 7d332a7c255b44589d54b1f583a2f9c4 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-7d332a7c255b44589d54b1f583a2f9c4
I20240503 01:59:38.209137 10672 tablet_copy_client.cc:308] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:38.209681 10774 tablet_copy_client.cc:783] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.209810 7391 tablet_copy_source_session.cc:205] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.209964 10774 tablet_copy_client.cc:647] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.210178 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:45331)
I20240503 01:59:38.210290 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:45331: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-c95d3c29bfbd4899b27b770bf9d3f4a6
I20240503 01:59:38.211231 7654 tablet_copy_source_session.cc:205] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.211582 10774 tablet_copy_client.cc:523] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.212522 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d332a7c255b44589d54b1f583a2f9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.213586 10673 ts_tablet_manager.cc:902] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.213676 10774 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.214082 10673 tablet_copy_client.cc:308] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.214905 10798 tablet_copy_client.cc:783] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.215071 10798 tablet_copy_client.cc:647] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.215374 10776 ts_tablet_manager.cc:902] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:38.215930 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.215986 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 6649e1feab5b40738003119a9122f07a from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:38.216094 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 6649e1feab5b40738003119a9122f07a from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-6649e1feab5b40738003119a9122f07a
I20240503 01:59:38.216967 7391 tablet_copy_source_session.cc:205] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.217793 10719 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.217986 10673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.219884 10673 tablet_copy_client.cc:783] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.220053 10673 tablet_copy_client.cc:647] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.224013 10672 tablet_copy_client.cc:783] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.224203 10672 tablet_copy_client.cc:647] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.233709 7239 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.235823 10926 tablet_replica.cc:330] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: stopping tablet replica
I20240503 01:59:38.236260 10926 raft_consensus.cc:2229] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 01:59:38.236697 10926 raft_consensus.cc:2258] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 01:59:38.237668 10776 tablet_copy_client.cc:308] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:38.238193 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:39201)
I20240503 01:59:38.238301 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:39201: session id = c778431b13784c7c81f3924b7f73e664-c95d3c29bfbd4899b27b770bf9d3f4a6
I20240503 01:59:38.239156 7654 tablet_copy_source_session.cc:205] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.240556 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.240770 10673 tablet_copy_client.cc:523] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.242218 10673 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.242165 7370 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:38.243216 10776 tablet_copy_client.cc:783] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.243430 10776 tablet_copy_client.cc:647] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.244058 7239 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.244511 10442 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.244870 9256 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.246953 10076 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.248970 10442 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.252111 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.253146 10672 tablet_copy_client.cc:523] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.253226 8994 raft_consensus.cc:1203] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.254819 10672 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.255115 9256 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.252681 7092 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.257912 10899 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.258263 10899 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.258522 10899 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.102s user 0.028s sys 0.000s
I20240503 01:59:38.258819 7239 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.261436 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:38.265137 10926 ts_tablet_manager.cc:1875] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.271762 10824 tablet_copy_client.cc:523] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.273471 10824 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.281021 10899 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.281317 10899 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.281492 10899 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.282511 10899 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.284307 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-527af716906b42729952db28e790bafd received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:38.284405 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-527af716906b42729952db28e790bafd on tablet 527af716906b42729952db28e790bafd with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.286564 10647 ts_tablet_manager.cc:902] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.286976 10647 tablet_copy_client.cc:308] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.287477 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 6649e1feab5b40738003119a9122f07a from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:38.287595 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 6649e1feab5b40738003119a9122f07a from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-6649e1feab5b40738003119a9122f07a
I20240503 01:59:38.288558 7391 tablet_copy_source_session.cc:205] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.289556 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.291887 10076 raft_consensus.cc:1050] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.292295 10647 tablet_copy_client.cc:783] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.292348 10076 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.292459 10647 tablet_copy_client.cc:647] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.293579 10837 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.293902 10776 tablet_copy_client.cc:523] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.293936 10837 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.294106 10837 ts_tablet_manager.cc:1367] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.253s user 0.023s sys 0.005s
W20240503 01:59:38.295078 7307 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.295367 10776 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.295632 8994 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.296042 9522 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.297262 7632 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.298128 10808 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.298394 10879 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.298419 10808 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.298637 10808 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.271s user 0.023s sys 0.006s
I20240503 01:59:38.299069 10808 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.299186 10808 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.299276 10808 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.299793 9522 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.300179 8994 raft_consensus.cc:1203] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.300666 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.301575 10442 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.302088 10442 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.302631 10926 ts_tablet_manager.cc:1888] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:38.302723 10926 log.cc:1196] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:38.303037 10926 ts_tablet_manager.cc:902] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:38.304351 10926 tablet_copy_client.cc:308] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:38.304909 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet be1f4d2622c6452b869f88a301a5e9bc from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:59397)
I20240503 01:59:38.304999 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet be1f4d2622c6452b869f88a301a5e9bc from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:59397: session id = 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:38.305013 8993 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.305838 10775 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.305956 7654 tablet_copy_source_session.cc:205] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.306149 10775 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.306298 10775 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.286s user 0.029s sys 0.000s
I20240503 01:59:38.306553 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.306717 10775 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.306916 10926 ts_tablet_manager.cc:1875] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 01:59:38.308624 7099 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.309415 10926 ts_tablet_manager.cc:1888] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 01:59:38.315482 7240 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.318279 7370 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.320792 10926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.323645 7632 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.323659 10837 raft_consensus.cc:348] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.323781 10837 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.323905 10837 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.324102 10077 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.324543 10926 tablet_copy_client.cc:783] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.324756 10926 tablet_copy_client.cc:647] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.324957 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.325970 10076 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.326583 10926 tablet_copy_client.cc:523] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.328198 10647 tablet_copy_client.cc:523] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.328424 10076 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.329641 10647 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:38.329771 10926 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.330904 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad24b47690b4263bdf4a9157db7c17f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.332559 7240 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.325330 9125 raft_consensus.cc:1203] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 01:59:38.333461 10775 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.330710 9124 raft_consensus.cc:1203] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.334779 8985 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.335750 10236 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.335819 10808 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240503 01:59:38.336198 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-81086ff737c84d9d9f04743d31c6151b received from {username='slave'} at 127.0.99.133:57905
I20240503 01:59:38.336275 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-81086ff737c84d9d9f04743d31c6151b on tablet 81086ff737c84d9d9f04743d31c6151b with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.336375 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.337280 8985 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.338012 9124 raft_consensus.cc:1203] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.339708 7096 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 6 to 7, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
W20240503 01:59:38.342252 10673 log.cc:924] Time spent T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Append to log took a long time: real 0.051s user 0.003s sys 0.000s
I20240503 01:59:38.333607 10775 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.347731 10775 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20240503 01:59:38.348261 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-f0539dded7554723b9e918e8cdfbab20 received from {username='slave'} at 127.0.99.134:45331
I20240503 01:59:38.348356 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-f0539dded7554723b9e918e8cdfbab20 on tablet f0539dded7554723b9e918e8cdfbab20 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.359066 10837 ts_tablet_manager.cc:1398] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.065s user 0.001s sys 0.000s
I20240503 01:59:38.360055 10798 tablet_copy_client.cc:523] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.360282 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-fb00fcf563b74243b7c955ddacf1d6cf received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:38.360378 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-fb00fcf563b74243b7c955ddacf1d6cf on tablet fb00fcf563b74243b7c955ddacf1d6cf with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.361593 10798 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:38.363054 10670 tablet_copy_client.cc:523] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.364460 10670 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.366371 10879 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.366572 10879 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.253s user 0.026s sys 0.002s
I20240503 01:59:38.367123 10879 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.367254 10879 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.367408 10879 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.367738 10879 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:38.368510 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-8ad24b47690b4263bdf4a9157db7c17f received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:38.368594 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-8ad24b47690b4263bdf4a9157db7c17f on tablet 8ad24b47690b4263bdf4a9157db7c17f with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.372973 10767 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.373791 9256 raft_consensus.cc:1203] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.375653 10719 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.376017 8985 raft_consensus.cc:1203] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.377488 10954 tablet_replica.cc:330] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:38.377912 10954 raft_consensus.cc:2229] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.378188 10954 raft_consensus.cc:2258] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.379745 10767 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.380028 10767 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.232s user 0.026s sys 0.004s
I20240503 01:59:38.380597 10767 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.380812 10767 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.373690 10879 ts_tablet_manager.cc:902] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:38.381716 10879 tablet_copy_client.cc:308] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:38.382205 7654 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet be1f4d2622c6452b869f88a301a5e9bc from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:39201)
I20240503 01:59:38.382302 7654 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet be1f4d2622c6452b869f88a301a5e9bc from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:39201: session id = c778431b13784c7c81f3924b7f73e664-be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:38.383173 7654 tablet_copy_source_session.cc:205] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.384760 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1f4d2622c6452b869f88a301a5e9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.387490 10879 tablet_copy_client.cc:783] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.387707 10879 tablet_copy_client.cc:647] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.380981 10767 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.388692 10767 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 01:59:38.389129 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-f0539dded7554723b9e918e8cdfbab20 received from {username='slave'} at 127.0.99.135:39201
I20240503 01:59:38.389201 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c778431b13784c7c81f3924b7f73e664-f0539dded7554723b9e918e8cdfbab20 on tablet f0539dded7554723b9e918e8cdfbab20 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.392304 10719 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.392580 10719 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.175s user 0.029s sys 0.000s
I20240503 01:59:38.393133 10719 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.393357 10719 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.393523 10719 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.394784 10699 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.394793 10719 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.395231 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-a0b1a4b2ede6493f9e4e094649b4b57d received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:38.395320 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-a0b1a4b2ede6493f9e4e094649b4b57d on tablet a0b1a4b2ede6493f9e4e094649b4b57d with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.398319 9387 raft_consensus.cc:1203] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.401525 10719 ts_tablet_manager.cc:902] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.401962 10719 tablet_copy_client.cc:308] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.402379 7391 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 6649e1feab5b40738003119a9122f07a from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:38.402493 7391 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 6649e1feab5b40738003119a9122f07a from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-6649e1feab5b40738003119a9122f07a
I20240503 01:59:38.403425 7391 tablet_copy_source_session.cc:205] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.404225 10879 tablet_copy_client.cc:523] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.412005 9387 raft_consensus.cc:1203] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 01:59:38.412642 10879 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:38.413192 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.419595 10719 tablet_copy_client.cc:783] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.419859 10719 tablet_copy_client.cc:647] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.441260 10647 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.441721 10777 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.442185 10954 ts_tablet_manager.cc:1875] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.442487 10647 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.442744 10647 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.113s user 0.028s sys 0.000s
I20240503 01:59:38.443787 10762 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.444123 10647 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.444353 10647 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.445220 10647 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.446283 10647 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.446775 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-6649e1feab5b40738003119a9122f07a received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:38.446849 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-6649e1feab5b40738003119a9122f07a on tablet 6649e1feab5b40738003119a9122f07a with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.450851 8908 raft_consensus.cc:1050] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:38.451361 8908 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:38.453292 7502 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.453830 7370 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.454373 10337 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.455947 10991 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.458091 10720 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.458446 10720 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.458707 10720 ts_tablet_manager.cc:1367] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.679s user 0.020s sys 0.007s
I20240503 01:59:38.459582 10720 raft_consensus.cc:348] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.459832 10720 raft_consensus.cc:729] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.459975 10720 consensus_queue.cc:260] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.469877 8985 raft_consensus.cc:1203] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.395164 10699 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:38.470957 10699 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.766s user 0.026s sys 0.000s
I20240503 01:59:38.471465 10699 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.471701 10699 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:38.471835 10699 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.472266 10699 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.472774 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-27c539b143674bdd8e3e5aaa4b78c1e9 received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:38.472855 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-27c539b143674bdd8e3e5aaa4b78c1e9 on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.479508 10954 ts_tablet_manager.cc:1888] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:38.479759 10954 log.cc:1196] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/97ffb59d3ab64d03aca5d30a7c3be1e1
I20240503 01:59:38.480435 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
I20240503 01:59:38.482671 7370 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.483443 10991 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.484447 7239 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.485222 10236 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.486611 7370 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.487016 7632 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:38.487797 7306 consensus_peers.cc:485] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9796308eacd94fb5ad3cb524aa590cfa. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.488011 9522 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.489014 7090 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:38.490222 7307 consensus_peers.cc:485] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.491595 7502 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.445571 9236 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f with delete_type TABLET_DATA_TOMBSTONED (TS c778431b13784c7c81f3924b7f73e664 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51440
I20240503 01:59:38.496299 10337 raft_consensus.cc:1050] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.455459 9256 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.496716 10337 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.497046 10991 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.464562 10879 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.497951 10879 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.498163 10879 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.000s
I20240503 01:59:38.498457 9256 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.498621 10879 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.498746 10879 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.498857 10879 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.499713 10879 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.500137 10762 tablet_bootstrap.cc:492] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.500191 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-be1f4d2622c6452b869f88a301a5e9bc received from {username='slave'} at 127.0.99.135:39201
I20240503 01:59:38.500286 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c778431b13784c7c81f3924b7f73e664-be1f4d2622c6452b869f88a301a5e9bc on tablet be1f4d2622c6452b869f88a301a5e9bc with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.500276 8985 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.500396 10762 ts_tablet_manager.cc:1367] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.348s user 0.026s sys 0.002s
I20240503 01:59:38.500838 10762 raft_consensus.cc:348] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.500963 10762 raft_consensus.cc:729] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.501057 10762 consensus_queue.cc:260] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.501361 10762 ts_tablet_manager.cc:1398] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.501526 10776 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.501744 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-cc68f31e6ef8411e96d46b2419bf9676 received from {username='slave'} at 127.0.99.135:39201
I20240503 01:59:38.501813 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c778431b13784c7c81f3924b7f73e664-cc68f31e6ef8411e96d46b2419bf9676 on tablet cc68f31e6ef8411e96d46b2419bf9676 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.501953 10776 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.502017 8908 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.502163 10776 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.207s user 0.025s sys 0.001s
I20240503 01:59:38.502624 10776 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.502756 10776 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.502861 10776 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.503140 10776 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.503710 7502 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.504195 8908 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.505280 7370 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.505676 8908 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.506662 8993 raft_consensus.cc:1203] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.510623 10442 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.513054 10673 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.513361 10673 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.513244 7632 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.513557 10673 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.271s user 0.026s sys 0.002s
I20240503 01:59:38.513536 7090 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.514037 10673 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.514171 10673 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.514317 10673 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.514621 10673 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.510784 11004 tablet_replica.cc:330] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: stopping tablet replica
I20240503 01:59:38.516429 11004 raft_consensus.cc:2229] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 01:59:38.517268 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-6649e1feab5b40738003119a9122f07a received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:38.517343 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-6649e1feab5b40738003119a9122f07a on tablet 6649e1feab5b40738003119a9122f07a with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.518152 11004 raft_consensus.cc:2258] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 01:59:38.518640 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-c95d3c29bfbd4899b27b770bf9d3f4a6 received from {username='slave'} at 127.0.99.135:39201
I20240503 01:59:38.518708 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c778431b13784c7c81f3924b7f73e664-c95d3c29bfbd4899b27b770bf9d3f4a6 on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.520893 8993 raft_consensus.cc:1203] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.521692 11004 ts_tablet_manager.cc:1875] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.524691 10769 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.525095 10769 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.525699 10769 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.337s user 0.021s sys 0.007s
I20240503 01:59:38.492511 10874 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.529906 10874 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.530237 10874 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.412s user 0.026s sys 0.004s
I20240503 01:59:38.530781 10874 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.531142 10874 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.526208 10720 ts_tablet_manager.cc:1398] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.067s user 0.001s sys 0.000s
I20240503 01:59:38.531878 10874 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.532408 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-cc68f31e6ef8411e96d46b2419bf9676 received from {username='slave'} at 127.0.99.134:45331
I20240503 01:59:38.532493 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-cc68f31e6ef8411e96d46b2419bf9676 on tablet cc68f31e6ef8411e96d46b2419bf9676 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.532930 10874 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.533433 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-a0b1a4b2ede6493f9e4e094649b4b57d received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:38.533506 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-a0b1a4b2ede6493f9e4e094649b4b57d on tablet a0b1a4b2ede6493f9e4e094649b4b57d with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.533636 7371 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.526877 10760 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.535392 9256 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.535425 10270 raft_consensus.cc:1050] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
W20240503 01:59:38.535641 7306 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.535838 10235 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.536013 10270 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.536808 10760 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.537099 10760 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.431s user 0.024s sys 0.004s
I20240503 01:59:38.537283 7240 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.537654 10760 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.537802 10672 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.538131 10672 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.538223 10076 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.538277 9125 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.538302 10672 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.284s user 0.023s sys 0.005s
I20240503 01:59:38.537844 10760 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.538697 10672 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.538815 10672 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.538978 10672 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.539036 10295 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.539263 10672 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.539435 7240 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.539451 10760 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.539816 10760 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.539862 10295 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.540344 10245 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.540596 10245 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.540607 7501 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.540724 10245 ts_tablet_manager.cc:1367] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.568s user 0.025s sys 0.001s
W20240503 01:59:38.540715 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.540982 10769 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
W20240503 01:59:38.541047 7568 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.541082 10769 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.541055 10245 raft_consensus.cc:348] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.541287 10245 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.541231 10769 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.541401 10245 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.541729 10245 ts_tablet_manager.cc:1398] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.542016 10295 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.542246 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-9796308eacd94fb5ad3cb524aa590cfa received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:38.542328 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-9796308eacd94fb5ad3cb524aa590cfa on tablet 9796308eacd94fb5ad3cb524aa590cfa with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.542578 10769 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 01:59:38.543084 11004 ts_tablet_manager.cc:1888] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:38.543943 10670 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.544050 7632 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.544165 11004 log.cc:1196] T 8ad24b47690b4263bdf4a9157db7c17f P c778431b13784c7c81f3924b7f73e664: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:38.544448 9522 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.544854 7085 catalog_manager.cc:4830] TS c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): tablet 8ad24b47690b4263bdf4a9157db7c17f (table rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]) successfully deleted
I20240503 01:59:38.546561 9522 raft_consensus.cc:1013] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer c778431b13784c7c81f3924b7f73e664: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.453594 9387 raft_consensus.cc:1203] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.478714 10719 tablet_copy_client.cc:523] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.547230 10670 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.547415 10670 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.183s user 0.025s sys 0.000s
I20240503 01:59:38.547405 10076 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.547988 10670 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.548091 10670 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.548174 10670 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.548519 10670 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.548626 10719 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:38.549898 10926 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.550180 10926 tablet_bootstrap.cc:492] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.550340 10926 ts_tablet_manager.cc:1367] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.221s user 0.026s sys 0.003s
I20240503 01:59:38.550312 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session 72577fce0313400ea862bf873738af54-527af716906b42729952db28e790bafd received from {username='slave'} at 127.0.99.136:50205
I20240503 01:59:38.550379 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session 72577fce0313400ea862bf873738af54-527af716906b42729952db28e790bafd on tablet 527af716906b42729952db28e790bafd with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.550371 11006 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.550745 10926 raft_consensus.cc:348] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.550858 10926 raft_consensus.cc:729] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.550956 10926 consensus_queue.cc:260] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.551431 9256 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.551524 7240 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.551896 7091 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.553149 10236 raft_consensus.cc:1050] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:38.553617 10236 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.544106 10777 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.554448 10777 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.548s user 0.023s sys 0.005s
I20240503 01:59:38.554944 10777 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.555130 10777 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.554991 7502 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.555296 10777 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.555892 10926 ts_tablet_manager.cc:1398] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:38.555924 10777 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.556998 10077 raft_consensus.cc:1050] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.557382 10077 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.557677 7239 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.558883 8985 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.560043 8993 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.560326 7091 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.560595 9386 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.561187 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.561686 9256 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:38.561776 7306 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.562005 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.562388 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.562691 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.563246 7654 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc received from {username='slave'} at 127.0.99.136:59397
I20240503 01:59:38.563335 7654 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 72577fce0313400ea862bf873738af54-be1f4d2622c6452b869f88a301a5e9bc on tablet be1f4d2622c6452b869f88a301a5e9bc with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.563669 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 72577fce0313400ea862bf873738af54-b4628afbfeff4f8bbaba5da1e89e4850 received from {username='slave'} at 127.0.99.136:59397
I20240503 01:59:38.563735 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 72577fce0313400ea862bf873738af54-b4628afbfeff4f8bbaba5da1e89e4850 on tablet b4628afbfeff4f8bbaba5da1e89e4850 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.565217 10295 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.566018 10442 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.566044 7239 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.566314 10442 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.567101 7240 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.569195 7239 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.569518 7503 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.569870 7370 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.572305 7096 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.573653 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.576797 9125 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.577152 10077 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.578294 9125 raft_consensus.cc:858] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
W20240503 01:59:38.578394 9125 raft_consensus.cc:1565] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.578914 7241 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.578881 9125 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.580452 7502 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.580565 7503 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.581084 10991 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.582466 8993 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.582868 11006 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.583650 7241 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.584201 7391 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-68791b3adccd4168b55a2da4423232d7 received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:38.584275 7391 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-68791b3adccd4168b55a2da4423232d7 on tablet 68791b3adccd4168b55a2da4423232d7 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.584494 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a432cd5fe20941d6bc73dae386d85170 received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:38.584553 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a432cd5fe20941d6bc73dae386d85170 on tablet a432cd5fe20941d6bc73dae386d85170 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.584617 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-c95d3c29bfbd4899b27b770bf9d3f4a6 received from {username='slave'} at 127.0.99.134:45331
I20240503 01:59:38.584681 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-c95d3c29bfbd4899b27b770bf9d3f4a6 on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.584750 10340 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.586480 8993 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.586833 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.587977 8908 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.584558 7370 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.589525 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18468345c7924d4f88366524b56e8ab3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.589682 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:38.591248 7438 consensus_peers.cc:485] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb00fcf563b74243b7c955ddacf1d6cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.592073 10759 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.592406 10759 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:38.449724 10729 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.592816 10729 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.592864 7503 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.593152 10729 ts_tablet_manager.cc:1367] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.406s user 0.027s sys 0.000s
I20240503 01:59:38.593155 7090 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.593758 10729 raft_consensus.cc:348] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.593961 10729 raft_consensus.cc:729] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.594164 10729 consensus_queue.cc:260] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.594605 10236 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.597177 10768 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.597391 10768 tablet_bootstrap.cc:492] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.597530 10768 ts_tablet_manager.cc:1367] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.404s user 0.025s sys 0.000s
I20240503 01:59:38.597976 10768 raft_consensus.cc:348] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.598088 10768 raft_consensus.cc:729] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.598156 10768 consensus_queue.cc:260] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.598948 10768 ts_tablet_manager.cc:1398] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:38.599319 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session 72577fce0313400ea862bf873738af54-3b15d48b2fa04269b98c6f801f54df4f received from {username='slave'} at 127.0.99.136:50205
I20240503 01:59:38.599385 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session 72577fce0313400ea862bf873738af54-3b15d48b2fa04269b98c6f801f54df4f on tablet 3b15d48b2fa04269b98c6f801f54df4f with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.600725 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet 18468345c7924d4f88366524b56e8ab3 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:38.600984 7634 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.601533 10442 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.601456 7240 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.602205 7634 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.603376 7099 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.603727 7634 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.605010 10335 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.605152 8985 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.605599 8993 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.606392 10335 raft_consensus.cc:1050] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.606839 10335 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.607013 7241 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
W20240503 01:59:38.609922 7438 consensus_peers.cc:485] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb00fcf563b74243b7c955ddacf1d6cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.611707 7370 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.612098 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.612356 8994 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.609468 7099 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added, now has a pending config: VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133), NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.612767 7370 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.613407 10339 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.614641 7240 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.615234 10335 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.615536 9387 raft_consensus.cc:1203] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.618237 10335 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.620287 9123 raft_consensus.cc:1203] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 01:59:38.620239 7240 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.620714 7370 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.625483 7099 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133), NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.638100 8994 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.640691 10774 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.640988 10774 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.641175 10774 ts_tablet_manager.cc:1367] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.428s user 0.023s sys 0.002s
I20240503 01:59:38.641640 10774 raft_consensus.cc:348] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.641783 10774 raft_consensus.cc:729] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.641907 10774 consensus_queue.cc:260] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.642176 10774 ts_tablet_manager.cc:1398] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.643134 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0a526f7fced4a04b4e0327f4656cf69 received from {username='slave'} at 127.0.99.134:56063
I20240503 01:59:38.643222 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-a0a526f7fced4a04b4e0327f4656cf69 on tablet a0a526f7fced4a04b4e0327f4656cf69 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.643597 10076 raft_consensus.cc:1013] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.644923 10076 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.645917 7503 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
W20240503 01:59:38.647871 7438 consensus_peers.cc:485] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb00fcf563b74243b7c955ddacf1d6cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.647503 9386 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.648306 7241 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.648401 8994 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.648782 11057 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.649082 11057 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.650723 11064 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.651068 7090 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.651842 7241 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.652014 10077 raft_consensus.cc:1050] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:38.652107 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.652375 10077 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.654635 7090 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 6 to 7, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.654844 8994 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.656162 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.656294 9386 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.656471 8985 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.656993 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.657181 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb00fcf563b74243b7c955ddacf1d6cf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.657277 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.657977 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet fb00fcf563b74243b7c955ddacf1d6cf with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:38.658041 8993 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:38.660310 7086 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.663615 10236 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.665712 10236 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.666178 11062 tablet_replica.cc:330] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:38.667153 11062 raft_consensus.cc:2229] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.668113 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.668831 7090 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.668862 8994 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.669715 11062 raft_consensus.cc:2258] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.672760 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.673831 9522 raft_consensus.cc:1050] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:38.674556 7241 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.675045 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.675276 9522 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.675479 10076 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.676039 10729 ts_tablet_manager.cc:1398] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.083s user 0.001s sys 0.000s
I20240503 01:59:38.677369 8994 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.677481 9125 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.678244 9385 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.678755 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-7d332a7c255b44589d54b1f583a2f9c4 received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:38.678828 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-7d332a7c255b44589d54b1f583a2f9c4 on tablet 7d332a7c255b44589d54b1f583a2f9c4 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.679275 7239 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.680133 10077 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.680480 10077 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:38.681375 7306 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.681469 7503 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.681878 9125 raft_consensus.cc:858] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
W20240503 01:59:38.681955 9125 raft_consensus.cc:1565] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.682226 9125 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.683434 9124 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.684687 7634 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.685355 9385 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.685616 10340 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.685806 11057 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.686252 10235 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.686580 10235 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:38.687099 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.687201 7241 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.687611 7503 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
W20240503 01:59:38.689929 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c18129344d124a5f8f688ec35ef69a55. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.690177 9385 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.690353 8994 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.690691 7633 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.690762 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.691066 10442 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.692624 10824 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.692858 7632 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.692968 10824 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:38.693167 10824 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.420s user 0.028s sys 0.000s
I20240503 01:59:38.693775 10442 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.693994 10824 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.694216 11064 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.694589 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.694875 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.695871 11063 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.696439 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.694101 10824 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:38.696579 7631 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.698218 10798 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.698586 10798 tablet_bootstrap.cc:492] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:38.698679 9124 raft_consensus.cc:1203] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.698820 10798 ts_tablet_manager.cc:1367] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.337s user 0.026s sys 0.003s
I20240503 01:59:38.699280 10798 raft_consensus.cc:348] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.699685 10798 raft_consensus.cc:729] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:38.699890 10798 consensus_queue.cc:260] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.699817 8993 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.700341 10798 ts_tablet_manager.cc:1398] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.701067 10824 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.702673 10824 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:38.703946 7091 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.704098 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-7d332a7c255b44589d54b1f583a2f9c4 received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:38.704191 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-7d332a7c255b44589d54b1f583a2f9c4 on tablet 7d332a7c255b44589d54b1f583a2f9c4 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:38.705431 8993 raft_consensus.cc:1203] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 01:59:38.705912 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session 72577fce0313400ea862bf873738af54-8ebe541161c84bd6b11bcd3d13d3bd2e received from {username='slave'} at 127.0.99.136:50205
I20240503 01:59:38.705988 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session 72577fce0313400ea862bf873738af54-8ebe541161c84bd6b11bcd3d13d3bd2e on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:38.705987 7631 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:38.706952 9522 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.707871 7240 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.708142 7503 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.708382 11033 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.708473 9123 raft_consensus.cc:1203] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.708554 10287 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.708868 9122 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.710114 11033 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.710435 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.710600 11033 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.711839 7631 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.712661 11062 ts_tablet_manager.cc:1875] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.713030 7238 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.713227 7503 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.714172 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.714752 7237 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.715745 7086 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 6: aborting the task: latest config opid_index 8; task opid_index 6
I20240503 01:59:38.716694 7091 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.717288 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4628afbfeff4f8bbaba5da1e89e4850 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.717543 9124 raft_consensus.cc:1203] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:38.719383 9124 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.719755 7092 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.719856 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f12a25a1db4650a6c37b5000a668c7 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:38.719383 9125 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.722208 8993 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.598690 9256 raft_consensus.cc:1203] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.601060 9254 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.693806 9253 raft_consensus.cc:1203] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.726294 9255 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.714830 10442 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.726641 7370 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.729662 9124 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.729877 9255 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.729884 7237 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.730106 10077 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.730315 10235 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.730485 10236 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 01:59:38.730772 7306 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.730871 9522 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.732229 9255 raft_consensus.cc:858] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
W20240503 01:59:38.732328 9255 raft_consensus.cc:1565] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.732646 9255 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.733197 10076 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.735076 10235 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.735131 8993 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.735318 7237 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.736362 10759 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.747s user 0.025s sys 0.005s
I20240503 01:59:38.737044 10759 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.737416 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a432cd5fe20941d6bc73dae386d85170 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.738277 9124 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.738446 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.739001 9255 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.740078 11064 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.741364 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.742151 7096 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.745393 8993 raft_consensus.cc:1203] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.745532 7094 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.746348 7633 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.748059 7099 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.739012 10759 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:38.750572 10759 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.750634 7632 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:38.752938 11064 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.753481 7370 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:38.753721 9255 raft_consensus.cc:1203] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.754081 11062 ts_tablet_manager.cc:1888] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:38.754277 10759 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240503 01:59:38.754745 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-5f35ccb9c21b4c10b9187596d2f6929c received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:38.754823 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-5f35ccb9c21b4c10b9187596d2f6929c on tablet 5f35ccb9c21b4c10b9187596d2f6929c with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.754760 7632 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.755038 11062 log.cc:1196] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/18468345c7924d4f88366524b56e8ab3
I20240503 01:59:38.756271 11062 tablet_replica.cc:330] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:38.756700 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet 18468345c7924d4f88366524b56e8ab3 (table rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]) successfully deleted
I20240503 01:59:38.756944 11062 raft_consensus.cc:2229] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.756995 9124 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.757253 7238 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:38.757257 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet a432cd5fe20941d6bc73dae386d85170 with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:38.757676 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.757844 7237 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.757937 9125 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:38.757661 7631 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.758361 9255 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:38.758638 8459 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.759743 9385 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:38.759780 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec179e5e683d4ba099c459331ea5e578. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.760251 10295 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.760924 7093 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.761479 10335 raft_consensus.cc:1050] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.761502 10295 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.761888 10335 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.762589 7238 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.763983 9125 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
W20240503 01:59:38.764228 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.764430 8994 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.765615 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.766459 8994 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.767102 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4628afbfeff4f8bbaba5da1e89e4850 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.767349 8459 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.768393 7240 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.768815 7096 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.768955 9125 raft_consensus.cc:1203] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.769560 7631 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.769976 9387 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.769940 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.770562 11062 raft_consensus.cc:2258] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.771221 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.771987 7237 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.772066 9387 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.757928 10442 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:38.763747 7370 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.773203 10295 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.774188 9255 raft_consensus.cc:1203] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.774413 11062 ts_tablet_manager.cc:1875] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.776217 10340 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.776798 11062 ts_tablet_manager.cc:1888] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 01:59:38.777169 7239 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.778210 10340 raft_consensus.cc:1013] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.778543 11095 tablet_replica.cc:330] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:38.778573 11102 tablet_replica.cc:330] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:38.779001 11095 raft_consensus.cc:2229] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.779268 11095 pending_rounds.cc:62] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Trying to abort 1 pending ops.
I20240503 01:59:38.780393 11102 raft_consensus.cc:2229] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.782692 11102 raft_consensus.cc:2258] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.783106 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet b4628afbfeff4f8bbaba5da1e89e4850 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:38.783248 9255 raft_consensus.cc:1203] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.783366 10954 tablet_replica.cc:330] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:38.776872 10235 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:38.783510 10954 raft_consensus.cc:2229] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.783735 10954 raft_consensus.cc:2258] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.783900 9522 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:38.784621 10954 ts_tablet_manager.cc:1875] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.786798 10442 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.787030 10954 ts_tablet_manager.cc:1888] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:38.787096 10954 log.cc:1196] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b4628afbfeff4f8bbaba5da1e89e4850
I20240503 01:59:38.787240 9253 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.788530 10339 raft_consensus.cc:1050] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:38.779430 11095 pending_rounds.cc:68] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023417665770106880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "71f12a25a1db4650a6c37b5000a668c7" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } } }
I20240503 01:59:38.788798 11095 raft_consensus.cc:2894] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 01:59:38.788914 11095 raft_consensus.cc:2866] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 01:59:38.788990 11095 raft_consensus.cc:2258] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.788969 10339 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:38.790132 7237 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.790688 7632 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.790688 9387 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.791114 11057 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.791468 11057 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.791867 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet b4628afbfeff4f8bbaba5da1e89e4850 (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
I20240503 01:59:38.791914 7240 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.792155 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a432cd5fe20941d6bc73dae386d85170 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.792289 11063 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.792750 7091 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.793812 9125 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.798060 11102 ts_tablet_manager.cc:1875] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.798254 11062 log.cc:1196] T fb00fcf563b74243b7c955ddacf1d6cf P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb00fcf563b74243b7c955ddacf1d6cf
I20240503 01:59:38.800387 7370 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.800964 9124 raft_consensus.cc:1203] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.802379 7238 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.799041 8459 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.805003 7096 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.807235 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet fb00fcf563b74243b7c955ddacf1d6cf (table rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]) successfully deleted
I20240503 01:59:38.809191 11095 ts_tablet_manager.cc:1875] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.809252 7370 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:38.809809 11102 ts_tablet_manager.cc:1888] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:38.809919 11102 log.cc:1196] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:38.811306 9125 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:38.811733 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet a432cd5fe20941d6bc73dae386d85170 (table rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]) successfully deleted
I20240503 01:59:38.813262 11063 raft_consensus.cc:1050] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.813426 9255 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:38.813730 11063 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.813863 10236 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:38.814086 9124 raft_consensus.cc:1203] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:38.814376 10236 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:38.816332 7370 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.816946 9386 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.816972 10335 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.817708 10335 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.817965 10442 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.818261 8993 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.818357 9125 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.818737 10335 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.820222 7631 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.820595 10335 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.822816 10339 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.823144 7241 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.825214 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a432cd5fe20941d6bc73dae386d85170 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.825354 8459 raft_consensus.cc:1050] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:38.825824 8459 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.825800 9255 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:38.825942 8994 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.827900 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.828291 10287 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.828552 7241 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.829030 9385 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.829500 11033 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.831172 11063 raft_consensus.cc:1050] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:38.831604 9384 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.831645 11063 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.832669 9367 tablet_service.cc:1514] Processing DeleteTablet for tablet b4628afbfeff4f8bbaba5da1e89e4850 with delete_type TABLET_DATA_TOMBSTONED (TS 72577fce0313400ea862bf873738af54 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41514
I20240503 01:59:38.833103 9366 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f12a25a1db4650a6c37b5000a668c7 with delete_type TABLET_DATA_TOMBSTONED (TS 72577fce0313400ea862bf873738af54 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41514
I20240503 01:59:38.833258 9365 tablet_service.cc:1514] Processing DeleteTablet for tablet a432cd5fe20941d6bc73dae386d85170 with delete_type TABLET_DATA_TOMBSTONED (TS 72577fce0313400ea862bf873738af54 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41514
I20240503 01:59:38.833505 10270 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:38.834108 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:38.834122 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec179e5e683d4ba099c459331ea5e578. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.834389 9385 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.834394 7091 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.835804 7092 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.837113 10287 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.837251 9385 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.837754 10335 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.837846 7631 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.839419 7371 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.839954 8985 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.839838 7240 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.840833 10339 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.843247 7095 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.843524 7241 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.844100 7633 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.844208 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.844588 10339 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.845232 7241 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.846580 10442 raft_consensus.cc:1050] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.846987 10442 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.849408 7502 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.849695 10335 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.852088 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet a432cd5fe20941d6bc73dae386d85170 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:38.852396 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.852291 7092 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.852615 11095 ts_tablet_manager.cc:1888] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 01:59:38.852780 11095 log.cc:1196] T 71f12a25a1db4650a6c37b5000a668c7 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/71f12a25a1db4650a6c37b5000a668c7
I20240503 01:59:38.853423 11095 tablet_replica.cc:330] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:38.853580 11095 raft_consensus.cc:2229] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.853787 11095 raft_consensus.cc:2258] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.854018 9384 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.856881 11127 tablet_replica.cc:330] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: stopping tablet replica
I20240503 01:59:38.857189 7503 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.852113 11086 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.857956 10442 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.858237 11127 raft_consensus.cc:2229] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 01:59:38.859016 8994 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.859210 11127 raft_consensus.cc:2258] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 01:59:38.859493 11064 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.859674 8985 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:38.860226 8985 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.862990 10339 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.863196 9387 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.863638 10335 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.863559 9387 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.866216 10076 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:38.866462 7096 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.867101 7237 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.872087 10442 raft_consensus.cc:1050] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.872623 11127 ts_tablet_manager.cc:1875] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.872618 10442 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:38.874398 7634 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.874632 9255 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.875012 7632 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.875169 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.875432 7239 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.875443 11127 ts_tablet_manager.cc:1888] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:38.875535 11127 log.cc:1196] T b4628afbfeff4f8bbaba5da1e89e4850 P 72577fce0313400ea862bf873738af54: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b4628afbfeff4f8bbaba5da1e89e4850
I20240503 01:59:38.875617 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.876111 7084 catalog_manager.cc:4830] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): tablet b4628afbfeff4f8bbaba5da1e89e4850 (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
I20240503 01:59:38.876842 10236 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.877902 8992 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.878399 10236 raft_consensus.cc:1050] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:38.878433 7632 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.878914 10236 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.878880 9387 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.879195 10442 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.881108 7633 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.881744 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.882503 7632 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.882858 9125 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.883169 7503 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.883940 10076 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.884275 10076 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:38.884510 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec179e5e683d4ba099c459331ea5e578. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.885700 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.886461 10335 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:38.886703 9253 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.886603 7633 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.887822 11033 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.888463 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.889492 9385 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.891902 10077 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.892985 9387 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.893447 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.893834 7086 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 527af716906b42729952db28e790bafd with cas_config_opid_index 7: aborting the task: latest config opid_index 8; task opid_index 7
I20240503 01:59:38.894513 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 71f12a25a1db4650a6c37b5000a668c7 (table rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]) successfully deleted
I20240503 01:59:38.895064 11033 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.895951 9125 raft_consensus.cc:1203] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.896173 7502 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.896565 11033 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.898703 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec179e5e683d4ba099c459331ea5e578 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.899034 10076 raft_consensus.cc:1013] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer c778431b13784c7c81f3924b7f73e664: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.899134 10992 raft_consensus.cc:1050] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:38.899291 8994 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.899434 7371 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.899679 10992 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.899636 7090 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.902038 8991 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.902928 7632 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.903442 10236 raft_consensus.cc:1013] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer c778431b13784c7c81f3924b7f73e664: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.903707 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.904116 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.905159 9125 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.905273 7099 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.905267 9522 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.905601 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.907440 7502 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.907505 9255 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.907930 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.907830 7633 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.910372 10236 raft_consensus.cc:1050] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:38.910547 7239 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.910753 10236 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:38.911355 9385 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.912855 7239 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.913372 10235 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.913614 7099 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240503 01:59:38.913913 8991 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.914484 9255 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.914927 10076 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.915200 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet ec179e5e683d4ba099c459331ea5e578 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:38.917865 7633 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.918246 10076 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.919235 7501 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:38.921145 11064 raft_consensus.cc:1050] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:38.921662 11064 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:38.923444 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.923840 11063 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.924235 9125 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.924726 7371 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.925099 8994 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.925310 11127 tablet_replica.cc:330] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: stopping tablet replica
I20240503 01:59:38.925477 11127 raft_consensus.cc:2229] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.925624 11127 pending_rounds.cc:62] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Trying to abort 1 pending ops.
I20240503 01:59:38.925653 11063 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.925693 11127 pending_rounds.cc:68] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023417666242617344 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "71f12a25a1db4650a6c37b5000a668c7" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } } }
I20240503 01:59:38.925846 11127 raft_consensus.cc:2894] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 01:59:38.926625 10337 raft_consensus.cc:1013] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: attempt to promote peer e648eca559ed4e5ab24a1b98755aa3eb: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.927506 10340 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.927814 10339 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.927953 7238 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.928206 9522 raft_consensus.cc:1013] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.930166 7634 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.932113 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 527af716906b42729952db28e790bafd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.932307 10340 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.932662 10335 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.934382 7240 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:38.934434 8990 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.935652 11086 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.935535 9387 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:38.925917 11127 raft_consensus.cc:2866] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 01:59:38.936612 11127 raft_consensus.cc:2258] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.936579 7372 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:38.938344 9522 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.938364 11064 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.938772 7098 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.939824 11127 ts_tablet_manager.cc:1875] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.939795 10335 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.940599 7631 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.940586 8985 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.942407 11127 ts_tablet_manager.cc:1888] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:38.942481 11127 log.cc:1196] T 71f12a25a1db4650a6c37b5000a668c7 P 72577fce0313400ea862bf873738af54: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/71f12a25a1db4650a6c37b5000a668c7
I20240503 01:59:38.942286 8990 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.943085 7084 catalog_manager.cc:4830] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): tablet 71f12a25a1db4650a6c37b5000a668c7 (table rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]) successfully deleted
W20240503 01:59:38.943364 7308 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad24b47690b4263bdf4a9157db7c17f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.944703 10340 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.944978 11063 raft_consensus.cc:1050] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:38.945340 11063 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.945899 8985 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.946658 10442 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.946974 7630 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.947214 9386 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.947731 7372 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.948113 10340 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.949167 9385 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.948891 7372 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.952040 10339 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.952860 7631 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.953178 8990 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.953620 10339 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.954329 8991 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.954807 7634 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.955214 10442 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.955468 10339 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:38.956684 11095 ts_tablet_manager.cc:1875] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:38.958652 11064 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.959335 7241 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.959477 7237 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:38.960101 9255 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.960677 9387 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.960939 7237 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.962785 10339 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.963436 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40260acad62242c3a72197d2dafae187 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:38.963807 7099 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.965023 9385 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.965157 7631 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
W20240503 01:59:38.967096 7086 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 5f35ccb9c21b4c10b9187596d2f6929c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240503 01:59:38.967195 10235 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.968791 9384 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.968896 10442 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.970307 10340 raft_consensus.cc:1013] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: attempt to promote peer e648eca559ed4e5ab24a1b98755aa3eb: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.970572 10991 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.970966 7096 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 5 to 6, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.971434 7239 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:38.971554 9124 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.973093 9122 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.975201 8992 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.975261 7240 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:38.976608 9255 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.977113 10992 raft_consensus.cc:1050] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.977144 7502 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.977454 10992 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.977586 9387 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.978677 10236 raft_consensus.cc:1050] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:38.979099 10236 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:38.979216 9122 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.979635 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.981398 7634 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.981511 10339 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.982266 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.982318 9387 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.982965 10729 ts_tablet_manager.cc:902] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:38.982924 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.983245 9125 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.983405 10729 tablet_copy_client.cc:308] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:38.983268 7634 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:38.983853 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 9796308eacd94fb5ad3cb524aa590cfa from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:38.983928 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 9796308eacd94fb5ad3cb524aa590cfa from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-9796308eacd94fb5ad3cb524aa590cfa
I20240503 01:59:38.984843 7392 tablet_copy_source_session.cc:205] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.985683 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9796308eacd94fb5ad3cb524aa590cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.985985 9122 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.985932 7371 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.986272 7501 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.986433 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.987071 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.988041 10077 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.988128 10729 tablet_copy_client.cc:783] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.988281 10729 tablet_copy_client.cc:647] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.988335 8992 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.988626 8985 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:38.989240 7629 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.990356 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.990739 10340 raft_consensus.cc:1050] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:38.991124 10340 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:38.991724 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.992906 9255 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:38.992961 7371 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.993286 8994 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.993484 9125 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:38.993602 10335 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.993808 10442 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:38.993854 11057 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.994402 7240 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:38.994606 11057 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:38.994910 8992 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:38.998185 10442 raft_consensus.cc:1013] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:38.999354 7092 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:39.000334 7084 catalog_manager.cc:4797] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): delete failed for tablet a432cd5fe20941d6bc73dae386d85170 because the tablet was not found. No further retry: Not found: Tablet not found: a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:39.001185 10287 raft_consensus.cc:1050] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.001407 10729 tablet_copy_client.cc:523] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.001641 10287 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:39.002851 10729 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:39.003639 9253 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.003705 9385 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.003871 7500 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.004382 10270 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.005931 10270 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.006692 10270 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.010139 10339 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.010408 7371 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.010859 10270 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.012403 7369 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.013911 7630 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.014561 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.014657 9125 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.014961 9254 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.015650 7632 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.017652 7629 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.018162 11064 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.018250 10337 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.018857 7629 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.019192 7633 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.019201 8993 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.020206 7629 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.023139 7092 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER, now has a pending config: VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.023183 7501 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.023725 7092 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.024420 7092 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.029793 8993 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.031502 11095 ts_tablet_manager.cc:1888] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 01:59:39.031661 11095 log.cc:1196] T a432cd5fe20941d6bc73dae386d85170 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:39.032253 11095 tablet_replica.cc:330] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.032449 11095 raft_consensus.cc:2229] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.032369 9125 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.032723 11095 raft_consensus.cc:2258] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.035794 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 40260acad62242c3a72197d2dafae187 with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:39.036010 11102 tablet_replica.cc:330] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:39.036152 11102 raft_consensus.cc:2229] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.036317 11102 raft_consensus.cc:2258] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.036623 10729 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.036962 10729 tablet_bootstrap.cc:492] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:39.037074 11102 ts_tablet_manager.cc:1875] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.037246 10729 ts_tablet_manager.cc:1367] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.005s
I20240503 01:59:39.037732 10729 raft_consensus.cc:348] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.037933 10729 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:39.038084 10729 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.038583 10729 ts_tablet_manager.cc:1398] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.039281 11102 ts_tablet_manager.cc:1888] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.039351 11102 log.cc:1196] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/40260acad62242c3a72197d2dafae187
I20240503 01:59:39.039350 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-9796308eacd94fb5ad3cb524aa590cfa received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:39.039419 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-9796308eacd94fb5ad3cb524aa590cfa on tablet 9796308eacd94fb5ad3cb524aa590cfa with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:39.040109 10270 raft_consensus.cc:1013] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: attempt to promote peer c778431b13784c7c81f3924b7f73e664: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.042230 11033 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.044353 9385 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.046732 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 40260acad62242c3a72197d2dafae187 (table rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]) successfully deleted
I20240503 01:59:39.046855 9253 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.047587 7094 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 6 to 7, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133). New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.047871 7094 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 5 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.048719 10719 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.050009 8459 raft_consensus.cc:1050] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:39.050462 8459 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.050912 7371 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:39.052677 9254 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 01:59:39.052934 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97ffb59d3ab64d03aca5d30a7c3be1e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.053445 7500 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.053560 7369 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.053947 11033 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.053860 7501 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.053963 7371 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.055284 9253 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.055670 10442 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.055984 11168 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.056515 11033 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.056818 11033 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.056767 7094 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 5 to 6, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.057523 10076 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.058094 7629 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.059610 9253 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.060346 9122 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.060837 11095 ts_tablet_manager.cc:1875] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.063383 11095 ts_tablet_manager.cc:1888] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 01:59:39.063580 11095 log.cc:1196] T ec179e5e683d4ba099c459331ea5e578 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ec179e5e683d4ba099c459331ea5e578
I20240503 01:59:39.063899 7090 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.052711 9385 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.065989 10719 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:39.065887 11033 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.066095 10287 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.068158 9253 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.068538 7371 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.068562 7501 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.070459 10719 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.522s user 0.029s sys 0.000s
I20240503 01:59:39.069620 9385 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.070995 7240 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.073136 10442 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.059746 9384 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.073971 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet a432cd5fe20941d6bc73dae386d85170 (table rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]) successfully deleted
I20240503 01:59:39.074146 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet ec179e5e683d4ba099c459331ea5e578 (table rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]) successfully deleted
I20240503 01:59:39.074672 9385 raft_consensus.cc:1203] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.074672 9236 tablet_service.cc:1514] Processing DeleteTablet for tablet 527af716906b42729952db28e790bafd with delete_type TABLET_DATA_TOMBSTONED (TS c778431b13784c7c81f3924b7f73e664 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51440
I20240503 01:59:39.075255 7237 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:39.076017 7241 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.078850 9122 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.079391 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.079993 9522 raft_consensus.cc:1050] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.080574 9522 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.081595 8993 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.082006 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.084056 10270 raft_consensus.cc:1050] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:39.084369 7629 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.084517 10270 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.084759 10442 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.086043 7500 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.086154 9385 raft_consensus.cc:1203] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:39.086369 9254 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.086952 10287 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.087410 9122 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.087767 7633 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.087924 11168 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.088193 11057 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.090232 9125 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.090287 7500 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.090178 7090 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.090821 7371 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.091099 9385 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.091118 7239 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.091327 9384 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.091629 11057 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.091794 11168 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.092430 9256 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.092684 8459 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.092986 8459 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.093086 8908 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.093180 8459 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.093789 7500 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.094215 8908 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.094439 8991 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.094931 10339 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.095469 10339 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.095853 8908 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.096016 10442 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.097959 7633 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.098520 9256 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.099162 7499 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.100873 9387 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.101343 7633 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.101723 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 527af716906b42729952db28e790bafd with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.103947 9122 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.106184 9385 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.107656 7092 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 5 to 6, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.108091 7502 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.108847 7501 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.109320 10442 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.109674 7632 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.110203 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet 66317a254eb644359f65610368d84d76 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.110198 10991 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.110225 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.110405 11062 tablet_replica.cc:330] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.110540 11062 raft_consensus.cc:2229] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.110715 11062 raft_consensus.cc:2258] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.111467 11062 ts_tablet_manager.cc:1875] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.111832 7501 raft_consensus.cc:858] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
W20240503 01:59:39.111959 7501 raft_consensus.cc:1565] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.112233 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.112593 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66317a254eb644359f65610368d84d76 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.113991 11062 ts_tablet_manager.cc:1888] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.114074 11062 log.cc:1196] T 66317a254eb644359f65610368d84d76 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/66317a254eb644359f65610368d84d76
I20240503 01:59:39.114740 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet 66317a254eb644359f65610368d84d76 (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.115224 7501 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.116899 10719 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.117375 10719 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:39.117566 10719 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.117790 10991 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.118130 10719 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.048s user 0.001s sys 0.000s
I20240503 01:59:39.118611 8993 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.119944 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-6649e1feab5b40738003119a9122f07a received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:39.120015 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-6649e1feab5b40738003119a9122f07a on tablet 6649e1feab5b40738003119a9122f07a with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:39.120352 9122 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.121668 8993 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.122468 7371 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.123673 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81086ff737c84d9d9f04743d31c6151b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.125334 11033 raft_consensus.cc:1013] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: attempt to promote peer c8ecb5fc8d804c9b9ca42ee4b2377f4e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.125730 9122 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.125715 10287 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.126137 11168 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:39.127664 7306 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68791b3adccd4168b55a2da4423232d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.129216 10759 ts_tablet_manager.cc:902] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:39.129302 7500 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.129679 10759 tablet_copy_client.cc:308] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:39.129654 11168 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.131672 7633 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.132431 11168 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.132329 9254 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.134533 7092 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.135775 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet fb00fcf563b74243b7c955ddacf1d6cf from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:36477)
I20240503 01:59:39.135890 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet fb00fcf563b74243b7c955ddacf1d6cf from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:36477: session id = c778431b13784c7c81f3924b7f73e664-fb00fcf563b74243b7c955ddacf1d6cf
I20240503 01:59:39.136996 7523 tablet_copy_source_session.cc:205] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.137943 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb00fcf563b74243b7c955ddacf1d6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.138876 10076 raft_consensus.cc:1013] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer c8ecb5fc8d804c9b9ca42ee4b2377f4e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.139767 7632 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.140440 10759 tablet_copy_client.cc:783] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.140594 10759 tablet_copy_client.cc:647] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.142163 7370 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.142432 7090 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.145112 7370 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.145376 7500 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.145745 10442 raft_consensus.cc:1050] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.146229 10442 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.146927 8990 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.148731 9125 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.148922 9385 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.149302 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.150102 7629 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.150463 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.151203 9122 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.151420 11125 raft_consensus.cc:1050] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.151590 10077 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.151983 11125 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.153650 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.153764 7500 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.154116 11033 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.154120 10077 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.154597 9254 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.154999 11033 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.156124 7093 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.157020 8990 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.157625 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.158568 9125 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.158933 11033 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.160161 7370 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.160558 10287 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.162158 9253 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.162542 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.162775 11169 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.163682 9253 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.165349 9253 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.167317 7099 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.170135 7372 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
W20240503 01:59:39.171195 7085 catalog_manager.cc:4797] TS c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): delete failed for tablet 527af716906b42729952db28e790bafd because the tablet was not found. No further retry: Not found: Tablet not found: 527af716906b42729952db28e790bafd
I20240503 01:59:39.171666 10287 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.172927 7090 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.173099 8993 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.173173 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 81086ff737c84d9d9f04743d31c6151b with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.173560 11168 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.173807 9255 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.174027 10954 tablet_replica.cc:330] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.174168 10954 raft_consensus.cc:2229] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.174371 10954 raft_consensus.cc:2258] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.174975 11168 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.177043 9125 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.177397 7372 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.177775 11168 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.179415 10954 ts_tablet_manager.cc:1875] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.179590 7239 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.180116 9253 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.180826 8991 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.181609 10954 ts_tablet_manager.cc:1888] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.181679 10954 log.cc:1196] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/81086ff737c84d9d9f04743d31c6151b
I20240503 01:59:39.182462 11086 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.182579 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 81086ff737c84d9d9f04743d31c6151b (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
I20240503 01:59:39.182741 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f35ccb9c21b4c10b9187596d2f6929c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.184209 7239 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.184401 7094 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 9 to 10, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.187474 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 527af716906b42729952db28e790bafd with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:39.187345 7500 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.187692 11095 tablet_replica.cc:330] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.187826 11095 raft_consensus.cc:2229] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.188002 11095 raft_consensus.cc:2258] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.188166 7090 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.189124 11095 ts_tablet_manager.cc:1875] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.190431 9254 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.190832 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:39.191356 11095 ts_tablet_manager.cc:1888] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 01:59:39.191439 11095 log.cc:1196] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/527af716906b42729952db28e790bafd
I20240503 01:59:39.192400 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 527af716906b42729952db28e790bafd (table rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]) successfully deleted
I20240503 01:59:39.193550 7094 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.203258 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:39.205327 9522 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.206239 11033 raft_consensus.cc:1050] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.206744 11033 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:39.207655 7629 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.208910 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.210207 10759 tablet_copy_client.cc:523] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.210824 10077 raft_consensus.cc:1050] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:39.211165 10077 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.211877 10759 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:39.212839 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.213200 9122 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.213598 10076 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.214422 7237 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.214753 10992 raft_consensus.cc:1050] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.215193 10992 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.215251 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.216579 8990 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.217132 7370 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.217583 10337 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.219019 9522 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.219362 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.219362 9122 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.219671 9125 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.219991 11006 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.220726 7630 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.221400 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.221844 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.222154 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.223569 7501 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.224047 9254 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.223912 9125 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.224493 7500 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.224598 8459 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.226305 11086 raft_consensus.cc:1050] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.226672 11086 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.227500 10992 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.227998 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 01:59:39.228781 7306 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68791b3adccd4168b55a2da4423232d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.228977 9124 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.229625 10077 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.230046 7371 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.230078 10077 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.230571 9124 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.231029 7502 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.232530 8990 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.233160 7503 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.233215 7629 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.234570 9125 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.234855 7502 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.235432 9387 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.236617 7500 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.236704 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 81086ff737c84d9d9f04743d31c6151b with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.237128 9387 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.237203 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.237702 9124 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.237800 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.238750 9387 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.240324 9387 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.240857 8990 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.241209 11064 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.242034 7630 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.242664 10335 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.242899 10335 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.244865 11168 raft_consensus.cc:1050] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.245357 11168 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.245683 7099 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 6 to 8, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.245929 11064 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.245945 7099 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added, now has a pending config: VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.246387 10270 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.247007 9124 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.247341 9385 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.247678 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.247685 10295 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.247795 10270 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.248979 7630 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.250119 7629 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.250263 11064 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.250560 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.250646 8459 raft_consensus.cc:1050] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:39.250720 11057 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.250821 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.250990 8459 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.252071 7502 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.252542 7634 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.253436 8990 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.253811 10077 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.254431 7634 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.255106 9124 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.255753 11063 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.256027 10339 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.257339 8993 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.258294 9385 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.258471 10335 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.258387 7630 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.259086 10077 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.260114 9253 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.260236 9123 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.260234 8992 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.260509 7369 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.260671 10295 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.261969 7630 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.262979 7502 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.263316 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 81086ff737c84d9d9f04743d31c6151b (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
I20240503 01:59:39.263474 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.263633 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 527af716906b42729952db28e790bafd with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.263916 10295 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.264132 10287 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.264238 9384 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.264271 9385 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.264544 11064 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.266481 11063 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.266597 9124 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.267285 8991 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.268939 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b15d48b2fa04269b98c6f801f54df4f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.269163 11033 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.269721 7630 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.270363 9384 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.272095 7368 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.273190 10076 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.273627 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f35ccb9c21b4c10b9187596d2f6929c with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:39.273831 11102 tablet_replica.cc:330] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:39.273952 11102 raft_consensus.cc:2229] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.274135 11102 raft_consensus.cc:2258] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.274503 11260 ts_tablet_manager.cc:902] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:39.275067 11260 tablet_copy_client.cc:308] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:39.275429 9121 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.275615 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet c18129344d124a5f8f688ec35ef69a55 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:39.275738 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet c18129344d124a5f8f688ec35ef69a55 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-c18129344d124a5f8f688ec35ef69a55
I20240503 01:59:39.276410 9384 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.276760 7523 tablet_copy_source_session.cc:205] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.276822 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.276953 7629 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.277511 9253 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.277550 11168 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.277760 10295 raft_consensus.cc:1013] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: attempt to promote peer c8ecb5fc8d804c9b9ca42ee4b2377f4e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.278054 11168 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.278561 11102 ts_tablet_manager.cc:1875] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.279470 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18129344d124a5f8f688ec35ef69a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.280841 8993 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.281149 11102 ts_tablet_manager.cc:1888] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.281226 11102 log.cc:1196] T 5f35ccb9c21b4c10b9187596d2f6929c P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f35ccb9c21b4c10b9187596d2f6929c
I20240503 01:59:39.282101 11260 tablet_copy_client.cc:783] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.282377 11260 tablet_copy_client.cc:647] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.282907 7503 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.285732 7095 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.285943 7237 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.287437 11168 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.290103 9385 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.296495 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0b1a4b2ede6493f9e4e094649b4b57d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.297039 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.297753 7093 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.298012 7237 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.298166 7093 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.299786 7237 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.302219 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.302459 11062 tablet_replica.cc:330] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.302609 11062 raft_consensus.cc:2229] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.302789 11062 raft_consensus.cc:2258] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.302820 7503 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.303586 11062 ts_tablet_manager.cc:1875] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.303776 9124 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.304222 10077 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.305056 9122 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.305465 10335 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.306164 11062 ts_tablet_manager.cc:1888] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.306236 11062 log.cc:1196] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/27c539b143674bdd8e3e5aaa4b78c1e9
I20240503 01:59:39.306175 9124 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.306972 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.307122 10335 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.308090 7095 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.308449 7095 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.308816 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.309227 9122 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.310554 7632 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.311026 11057 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.313238 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b15d48b2fa04269b98c6f801f54df4f with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.313720 7632 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.314158 7503 consensus_queue.cc:237] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.314363 11062 tablet_replica.cc:330] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.314491 11062 raft_consensus.cc:2229] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.314719 11062 raft_consensus.cc:2258] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.314992 7630 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.316563 8993 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.316428 9253 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.317195 9385 raft_consensus.cc:1261] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.317653 11057 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.318015 11057 consensus_queue.cc:1035] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.319226 7095 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.320013 10992 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.321509 11033 raft_consensus.cc:1013] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: attempt to promote peer c778431b13784c7c81f3924b7f73e664: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.322355 10335 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.323060 9385 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.323832 9122 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.325134 8993 raft_consensus.cc:2932] T 3b15d48b2fa04269b98c6f801f54df4f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.326412 7503 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.327706 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b15d48b2fa04269b98c6f801f54df4f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.328776 11062 ts_tablet_manager.cc:1875] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.328900 7098 catalog_manager.cc:5484] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.331004 11062 ts_tablet_manager.cc:1888] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.331147 11062 log.cc:1196] T 3b15d48b2fa04269b98c6f801f54df4f P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b15d48b2fa04269b98c6f801f54df4f
I20240503 01:59:39.331527 7092 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 9 to 10, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.333112 11260 tablet_copy_client.cc:523] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.334641 11260 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:39.335398 8974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.335696 10270 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.336741 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet 3b15d48b2fa04269b98c6f801f54df4f (table rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]) successfully deleted
I20240503 01:59:39.338181 9254 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.340837 9122 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.344327 7098 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.344437 7370 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.346864 8459 raft_consensus.cc:1050] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.347268 8459 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
W20240503 01:59:39.347855 11125 consensus_queue.cc:1159] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c778431b13784c7c81f3924b7f73e664" responder_term: 1 status { last_received { term: 1 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 6 } }
I20240503 01:59:39.349243 9254 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.349822 11125 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.349685 7502 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.351853 7372 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.353343 10295 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.353446 7370 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.355291 9122 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.357813 10270 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.363121 10270 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.365020 9522 raft_consensus.cc:1050] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e to VOTER
I20240503 01:59:39.365732 9254 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.365873 9522 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.366040 9122 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.367841 9385 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.368228 10442 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.368355 7502 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.370374 7502 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.370432 7237 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.370831 10077 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.371520 9122 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.372246 10077 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.372505 10077 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.373306 7370 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.376732 8974 tablet_service.cc:1514] Processing DeleteTablet for tablet 527af716906b42729952db28e790bafd with delete_type TABLET_DATA_TOMBSTONED (TS e648eca559ed4e5ab24a1b98755aa3eb not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52888
I20240503 01:59:39.376732 8973 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with delete_type TABLET_DATA_TOMBSTONED (TS e648eca559ed4e5ab24a1b98755aa3eb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52888
I20240503 01:59:39.383759 7240 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.385704 7370 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.386087 8908 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.387281 9124 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.387786 8908 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.389585 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0a526f7fced4a04b4e0327f4656cf69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.390182 7370 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.390491 11187 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.391752 9124 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.394107 7098 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.394584 7095 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.396576 7631 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.397204 11169 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.398324 7631 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.400146 7631 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.401284 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet a0a526f7fced4a04b4e0327f4656cf69 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.401495 10954 tablet_replica.cc:330] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.401633 10954 raft_consensus.cc:2229] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.401777 10954 raft_consensus.cc:2258] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.403096 7633 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.403867 10954 ts_tablet_manager.cc:1875] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.404083 7631 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.404536 9384 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.405089 10287 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.406144 10954 ts_tablet_manager.cc:1888] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.406219 10954 log.cc:1196] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a0a526f7fced4a04b4e0327f4656cf69
I20240503 01:59:39.407934 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.409559 11288 tablet_replica.cc:330] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: stopping tablet replica
I20240503 01:59:39.409893 11288 raft_consensus.cc:2229] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.410208 11288 raft_consensus.cc:2258] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.410516 11278 ts_tablet_manager.cc:902] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:39.410769 10077 raft_consensus.cc:1013] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.411060 11278 tablet_copy_client.cc:308] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:39.411049 9385 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.411437 10442 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.411706 11125 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.412210 11260 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.412494 11260 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:39.412667 11260 ts_tablet_manager.cc:1367] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.078s user 0.023s sys 0.000s
I20240503 01:59:39.412894 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.413280 10442 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.413532 11260 raft_consensus.cc:348] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:39.413643 11260 raft_consensus.cc:729] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:39.413700 9254 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.413741 11260 consensus_queue.cc:260] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:39.413919 9255 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.414100 11260 ts_tablet_manager.cc:1398] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:39.414077 8459 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.414604 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-c18129344d124a5f8f688ec35ef69a55 received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:39.414678 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-c18129344d124a5f8f688ec35ef69a55 on tablet c18129344d124a5f8f688ec35ef69a55 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:39.415975 8459 raft_consensus.cc:1050] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:39.416302 8459 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.416936 9125 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.417629 7500 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.419600 7372 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.420001 7371 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:39.420394 7500 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.420763 8993 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.421761 10235 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.422756 9255 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.423241 10270 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.423682 9125 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.423702 10270 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.423986 11125 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.425801 8459 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.426157 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet a0a526f7fced4a04b4e0327f4656cf69 (table rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]) successfully deleted
I20240503 01:59:39.426579 10759 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.426730 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.426880 10759 tablet_bootstrap.cc:492] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:39.426877 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:39.426934 10954 tablet_replica.cc:330] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.427058 10759 ts_tablet_manager.cc:1367] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.215s user 0.026s sys 0.000s
I20240503 01:59:39.427065 10954 raft_consensus.cc:2229] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.427304 10954 raft_consensus.cc:2258] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.427577 10759 raft_consensus.cc:348] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:39.427706 10759 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:39.427803 10759 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:39.428066 10759 ts_tablet_manager.cc:1398] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:39.428591 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-fb00fcf563b74243b7c955ddacf1d6cf received from {username='slave'} at 127.0.99.135:36477
I20240503 01:59:39.428658 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session c778431b13784c7c81f3924b7f73e664-fb00fcf563b74243b7c955ddacf1d6cf on tablet fb00fcf563b74243b7c955ddacf1d6cf with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:39.430166 8459 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.431330 7500 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.432487 9255 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.434767 9125 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.434856 11086 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.436792 10954 ts_tablet_manager.cc:1875] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.437299 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.437721 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.438162 9385 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.439218 10954 ts_tablet_manager.cc:1888] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 01:59:39.439301 10954 log.cc:1196] T a0b1a4b2ede6493f9e4e094649b4b57d P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:39.441794 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.443815 9385 raft_consensus.cc:1203] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:39.444388 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.444948 11125 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.445677 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet a0b1a4b2ede6493f9e4e094649b4b57d (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.445852 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6649e1feab5b40738003119a9122f07a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.447474 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 8ad24b47690b4263bdf4a9157db7c17f from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:53659)
I20240503 01:59:39.447286 11169 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.447559 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 8ad24b47690b4263bdf4a9157db7c17f from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:53659: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:39.447801 7237 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.448141 9125 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.448493 7392 tablet_copy_source_session.cc:205] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.449060 7372 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.452149 7093 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.452472 7093 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.453629 11125 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.454155 9385 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.456269 9522 raft_consensus.cc:1050] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.456647 9522 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.458695 9384 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.459092 10442 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.461478 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:39.462428 7240 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.462869 10235 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.466701 10235 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.467082 10270 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.469008 7240 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.471019 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:39.471438 9254 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.472182 7093 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.473879 11288 ts_tablet_manager.cc:1875] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.476595 11288 ts_tablet_manager.cc:1888] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 01:59:39.476684 11288 log.cc:1196] T 527af716906b42729952db28e790bafd P e648eca559ed4e5ab24a1b98755aa3eb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/527af716906b42729952db28e790bafd
I20240503 01:59:39.476962 9384 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.477238 11288 tablet_replica.cc:330] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: stopping tablet replica
I20240503 01:59:39.477295 7083 catalog_manager.cc:4830] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): tablet 527af716906b42729952db28e790bafd (table rebalance_test_table_8 [id=325e5f6cef5d4f3687726b6b27765419]) successfully deleted
I20240503 01:59:39.477384 11288 raft_consensus.cc:2229] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.477531 11288 raft_consensus.cc:2258] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.478298 11288 ts_tablet_manager.cc:1875] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.480170 11288 ts_tablet_manager.cc:1888] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.480234 11288 log.cc:1196] T 27c539b143674bdd8e3e5aaa4b78c1e9 P e648eca559ed4e5ab24a1b98755aa3eb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/27c539b143674bdd8e3e5aaa4b78c1e9
I20240503 01:59:39.480959 11304 ts_tablet_manager.cc:902] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:39.480924 7083 catalog_manager.cc:4830] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.481068 9253 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.481441 11304 tablet_copy_client.cc:308] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:39.481458 10295 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.483875 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet ec179e5e683d4ba099c459331ea5e578 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e ({username='slave'} at 127.0.99.134:45331)
I20240503 01:59:39.483990 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet ec179e5e683d4ba099c459331ea5e578 from peer c8ecb5fc8d804c9b9ca42ee4b2377f4e at {username='slave'} at 127.0.99.134:45331: session id = c8ecb5fc8d804c9b9ca42ee4b2377f4e-ec179e5e683d4ba099c459331ea5e578
I20240503 01:59:39.484833 7653 tablet_copy_source_session.cc:205] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.485702 11304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.488118 11304 tablet_copy_client.cc:783] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.488075 7370 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.488272 11304 tablet_copy_client.cc:647] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.489392 9253 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.489447 8991 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.489765 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.489675 11033 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.490084 11168 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.490054 9253 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.490330 11168 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.491914 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b15d48b2fa04269b98c6f801f54df4f with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:39.491966 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 5f35ccb9c21b4c10b9187596d2f6929c (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.492126 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be1f4d2622c6452b869f88a301a5e9bc with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.492249 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0539dded7554723b9e918e8cdfbab20 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.492333 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.492698 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 6649e1feab5b40738003119a9122f07a with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:39.492590 7500 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.494637 7371 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.495409 9124 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.496881 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.496899 7500 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.497821 10442 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.497937 9254 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.498431 7503 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.499223 11278 tablet_copy_client.cc:783] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.499408 11278 tablet_copy_client.cc:647] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.499475 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.500368 10076 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.500640 9124 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.501010 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f35ccb9c21b4c10b9187596d2f6929c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.501117 9123 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.502810 9122 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.502952 8993 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.503656 8459 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.504104 11102 tablet_replica.cc:330] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:39.504263 11102 raft_consensus.cc:2229] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.504477 11102 raft_consensus.cc:2258] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.505332 9122 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.505604 9385 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.508169 10236 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.508543 11102 ts_tablet_manager.cc:1875] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.509912 9125 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.511495 7370 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.512235 11278 tablet_copy_client.cc:523] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.513792 11278 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:39.515935 11102 ts_tablet_manager.cc:1888] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.516100 11102 log.cc:1196] T 3b15d48b2fa04269b98c6f801f54df4f P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b15d48b2fa04269b98c6f801f54df4f
I20240503 01:59:39.515889 7096 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 5 to 6, c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) changed from NON_VOTER to VOTER, now has a pending config: VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.516724 11102 tablet_replica.cc:330] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:39.516885 11102 raft_consensus.cc:2229] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.517114 11102 raft_consensus.cc:2258] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.517846 11102 ts_tablet_manager.cc:1875] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.523011 8973 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:52888
I20240503 01:59:39.523452 7083 catalog_manager.cc:4830] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): tablet 27c539b143674bdd8e3e5aaa4b78c1e9 (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.523533 7502 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.524941 7098 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130), VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.525498 7098 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.525858 7098 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.527992 9254 raft_consensus.cc:1203] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:39.529460 11102 ts_tablet_manager.cc:1888] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 01:59:39.529620 11102 log.cc:1196] T 6649e1feab5b40738003119a9122f07a P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6649e1feab5b40738003119a9122f07a
I20240503 01:59:39.530282 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 3b15d48b2fa04269b98c6f801f54df4f (table rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]) successfully deleted
I20240503 01:59:39.530433 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 6649e1feab5b40738003119a9122f07a (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.530848 7631 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.532217 9125 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 01:59:39.532413 7084 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet f0539dded7554723b9e918e8cdfbab20 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
W20240503 01:59:39.532610 7084 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet be1f4d2622c6452b869f88a301a5e9bc with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
I20240503 01:59:39.532824 11125 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.532922 7371 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.533270 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.533838 9254 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.534948 7092 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.535070 11125 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.535769 10270 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.537256 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc68f31e6ef8411e96d46b2419bf9676 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.537789 9125 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.537946 7371 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.543933 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f35ccb9c21b4c10b9187596d2f6929c with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:39.544088 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet be1f4d2622c6452b869f88a301a5e9bc with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.544031 7371 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.544159 11095 tablet_replica.cc:330] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.544288 11095 raft_consensus.cc:2229] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.544443 11095 raft_consensus.cc:2258] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.545156 11095 ts_tablet_manager.cc:1875] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.545866 7237 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.546787 7633 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.547207 9522 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.547425 11169 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.547645 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet f0539dded7554723b9e918e8cdfbab20 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.547818 11062 tablet_replica.cc:330] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.547959 11062 raft_consensus.cc:2229] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.548139 11062 raft_consensus.cc:2258] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.549037 7631 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.549124 10954 tablet_replica.cc:330] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.549275 10954 raft_consensus.cc:2229] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.549436 7632 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.549504 10954 raft_consensus.cc:2258] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.550676 9385 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.550909 11062 ts_tablet_manager.cc:1875] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.551000 9522 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.551724 10954 ts_tablet_manager.cc:1875] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.552000 7092 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.553210 11062 ts_tablet_manager.cc:1888] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.553275 11062 log.cc:1196] T f0539dded7554723b9e918e8cdfbab20 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0539dded7554723b9e918e8cdfbab20
I20240503 01:59:39.553609 8459 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.554041 7237 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.554455 7633 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.554738 10954 ts_tablet_manager.cc:1888] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.554812 10954 log.cc:1196] T be1f4d2622c6452b869f88a301a5e9bc P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:39.554930 7632 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.556025 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet be1f4d2622c6452b869f88a301a5e9bc (table rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]) successfully deleted
I20240503 01:59:39.556214 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad24b47690b4263bdf4a9157db7c17f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.556414 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet f0539dded7554723b9e918e8cdfbab20 (table rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]) successfully deleted
I20240503 01:59:39.557399 7099 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 8 to 9, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.557641 8459 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.557976 9385 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.558250 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.558924 9255 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.559410 9255 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.560287 9385 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.560258 10295 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.560617 10295 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.560850 11169 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.561412 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.547513 11095 ts_tablet_manager.cc:1888] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.562108 11095 log.cc:1196] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f35ccb9c21b4c10b9187596d2f6929c
I20240503 01:59:39.562193 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.562940 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 5f35ccb9c21b4c10b9187596d2f6929c (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.564451 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet cc68f31e6ef8411e96d46b2419bf9676 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.564468 9255 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.564698 11169 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.566824 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.567826 9254 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.568706 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be1f4d2622c6452b869f88a301a5e9bc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.569105 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f with delete_type TABLET_DATA_TOMBSTONED (TS c8ecb5fc8d804c9b9ca42ee4b2377f4e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34354
I20240503 01:59:39.570303 9253 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.570322 7237 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.570245 7090 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.570729 8993 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.570703 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.571171 9385 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.571406 7633 consensus_queue.cc:237] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.571408 7094 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.572070 9522 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.572304 7237 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.573500 10287 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 01:59:39.573655 7086 catalog_manager.cc:4810] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): delete failed for tablet 8ad24b47690b4263bdf4a9157db7c17f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad24b47690b4263bdf4a9157db7c17f already in progress: copying tablet
I20240503 01:59:39.574479 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6649e1feab5b40738003119a9122f07a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.575172 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.575232 7500 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.575393 9125 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.575841 8993 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.576385 9253 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.576385 9125 raft_consensus.cc:1261] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.576740 8459 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.576797 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.577261 10287 consensus_queue.cc:1035] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.577764 10295 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.578001 10295 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.578330 8459 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.578763 9253 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.580016 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc68f31e6ef8411e96d46b2419bf9676 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.581264 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet be1f4d2622c6452b869f88a301a5e9bc with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.582263 11062 tablet_replica.cc:330] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.582418 11062 raft_consensus.cc:2229] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.582645 11062 raft_consensus.cc:2258] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.583402 10295 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.584718 7092 catalog_manager.cc:5484] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.585335 11062 ts_tablet_manager.cc:1875] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.585978 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.586434 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6649e1feab5b40738003119a9122f07a with delete_type TABLET_DATA_TOMBSTONED (TS c8ecb5fc8d804c9b9ca42ee4b2377f4e not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34354
I20240503 01:59:39.586426 9255 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.588146 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:39.588551 10287 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.590082 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0539dded7554723b9e918e8cdfbab20 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.590421 9385 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:39.590886 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:39.592738 7500 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.592855 9522 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.593262 9385 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.595780 9122 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.596086 9254 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.597422 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet cc68f31e6ef8411e96d46b2419bf9676 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.597712 9125 raft_consensus.cc:2932] T cc68f31e6ef8411e96d46b2419bf9676 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.598209 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:39.598629 10236 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:39.599342 7092 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.599720 7092 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.599956 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.601960 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6649e1feab5b40738003119a9122f07a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.603286 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet f0539dded7554723b9e918e8cdfbab20 with delete_type TABLET_DATA_TOMBSTONED (TS c8ecb5fc8d804c9b9ca42ee4b2377f4e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34354
I20240503 01:59:39.603497 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35818
I20240503 01:59:39.603763 7090 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 11 to 12, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:39.603881 7502 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.607080 11062 ts_tablet_manager.cc:1888] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.607174 11062 log.cc:1196] T be1f4d2622c6452b869f88a301a5e9bc P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/be1f4d2622c6452b869f88a301a5e9bc
I20240503 01:59:39.607652 11062 tablet_replica.cc:330] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.607769 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet be1f4d2622c6452b869f88a301a5e9bc (table rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]) successfully deleted
I20240503 01:59:39.609503 11062 raft_consensus.cc:2229] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.609794 11062 raft_consensus.cc:2258] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.613560 11260 ts_tablet_manager.cc:902] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:39.614378 11260 tablet_copy_client.cc:308] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:39.615006 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:38831)
I20240503 01:59:39.615118 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:38831: session id = e648eca559ed4e5ab24a1b98755aa3eb-97ffb59d3ab64d03aca5d30a7c3be1e1
I20240503 01:59:39.615676 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.616194 11062 ts_tablet_manager.cc:1875] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.616227 7392 tablet_copy_source_session.cc:205] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.618585 11062 ts_tablet_manager.cc:1888] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.618839 11317 tablet_replica.cc:330] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: stopping tablet replica
I20240503 01:59:39.619290 11317 raft_consensus.cc:2229] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.619673 11317 raft_consensus.cc:2258] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.621058 11317 ts_tablet_manager.cc:1875] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.623339 11317 ts_tablet_manager.cc:1888] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 01:59:39.623515 11062 log.cc:1196] T cc68f31e6ef8411e96d46b2419bf9676 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc68f31e6ef8411e96d46b2419bf9676
I20240503 01:59:39.608502 7370 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.625602 7632 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.625834 11062 tablet_replica.cc:330] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 01:59:39.625962 11062 raft_consensus.cc:2229] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.626103 11062 raft_consensus.cc:2258] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.626804 11062 ts_tablet_manager.cc:1875] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.627826 10954 tablet_replica.cc:330] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.628098 10954 raft_consensus.cc:2229] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.628149 10236 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.628366 10954 raft_consensus.cc:2258] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.628842 11317 log.cc:1196] T 6649e1feab5b40738003119a9122f07a P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/6649e1feab5b40738003119a9122f07a
I20240503 01:59:39.629289 10954 ts_tablet_manager.cc:1875] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.629519 11317 tablet_replica.cc:330] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: stopping tablet replica
I20240503 01:59:39.629491 7086 catalog_manager.cc:4830] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): tablet 6649e1feab5b40738003119a9122f07a (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.629709 11317 raft_consensus.cc:2229] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.629920 11317 raft_consensus.cc:2258] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.630676 9385 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.631227 10236 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.632277 10954 ts_tablet_manager.cc:1888] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.632789 7633 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:39.633188 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.633986 11317 ts_tablet_manager.cc:1875] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.634037 9522 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.635115 11062 ts_tablet_manager.cc:1888] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.635531 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 6649e1feab5b40738003119a9122f07a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 11 (current committed config index is 12)) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:39.636279 11095 tablet_replica.cc:330] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.636497 11095 raft_consensus.cc:2229] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.636771 11095 raft_consensus.cc:2258] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.637084 7632 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.637759 11095 ts_tablet_manager.cc:1875] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.638942 9385 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.630478 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet cc68f31e6ef8411e96d46b2419bf9676 (table rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]) successfully deleted
I20240503 01:59:39.640236 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0b1a4b2ede6493f9e4e094649b4b57d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.640671 11095 ts_tablet_manager.cc:1888] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 01:59:39.640794 11095 log.cc:1196] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6649e1feab5b40738003119a9122f07a
I20240503 01:59:39.641413 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 6649e1feab5b40738003119a9122f07a (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.641978 11062 log.cc:1196] T c95d3c29bfbd4899b27b770bf9d3f4a6 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c95d3c29bfbd4899b27b770bf9d3f4a6
I20240503 01:59:39.643312 9125 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.643872 9255 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.643908 11317 ts_tablet_manager.cc:1888] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.643968 11317 log.cc:1196] T f0539dded7554723b9e918e8cdfbab20 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f0539dded7554723b9e918e8cdfbab20
I20240503 01:59:39.644027 7090 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.644189 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.645004 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.646363 7086 catalog_manager.cc:4830] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): tablet f0539dded7554723b9e918e8cdfbab20 (table rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]) successfully deleted
I20240503 01:59:39.646726 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d with delete_type TABLET_DATA_TOMBSTONED (TS c8ecb5fc8d804c9b9ca42ee4b2377f4e not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34354
I20240503 01:59:39.646724 11125 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.647069 11317 tablet_replica.cc:330] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: stopping tablet replica
I20240503 01:59:39.647248 11317 raft_consensus.cc:2229] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.647456 11317 raft_consensus.cc:2258] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.648705 11317 ts_tablet_manager.cc:1875] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.649508 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.650086 9125 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.651177 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:39.652196 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet c95d3c29bfbd4899b27b770bf9d3f4a6 (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.652873 9255 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:39.651124 7095 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.655499 10076 raft_consensus.cc:1050] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:39.655985 10076 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.658030 9385 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.658445 7237 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.658540 9522 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.659034 9253 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.659930 11169 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.660260 11169 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.660416 9254 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:39.660786 10236 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:39.662776 9255 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.663292 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:39.666589 11317 ts_tablet_manager.cc:1888] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 01:59:39.666725 11317 log.cc:1196] T a0b1a4b2ede6493f9e4e094649b4b57d P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:39.667222 7370 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.668696 7632 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:39.669118 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:39.670147 9385 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:39.670560 9522 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:39.672335 10235 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.674643 9385 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.675693 7632 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.676360 7086 catalog_manager.cc:4830] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): tablet a0b1a4b2ede6493f9e4e094649b4b57d (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.677417 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0b1a4b2ede6493f9e4e094649b4b57d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.678028 9522 raft_consensus.cc:1013] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 72577fce0313400ea862bf873738af54: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:39.677613 7095 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 11 to 12, VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.682159 10235 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.682574 7237 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.683333 7633 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.683652 9236 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b1a4b2ede6493f9e4e094649b4b57d with delete_type TABLET_DATA_TOMBSTONED (TS c778431b13784c7c81f3924b7f73e664 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:51440
I20240503 01:59:39.684245 10235 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.686067 9384 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.686308 7090 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.686434 7633 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.688689 11168 raft_consensus.cc:1050] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.689110 11168 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.690526 9253 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.691970 7237 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.692574 10076 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.692709 9384 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.693377 7633 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.693708 10076 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.694209 10077 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.696470 11169 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.697351 11260 tablet_copy_client.cc:783] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.697585 11260 tablet_copy_client.cc:647] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.698050 9255 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.697958 9385 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.699985 10236 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.700199 7237 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.700989 9255 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.702298 7093 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.703030 11340 ts_tablet_manager.cc:902] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:39.703694 11340 tablet_copy_client.cc:308] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:39.703737 7633 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.705593 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 68791b3adccd4168b55a2da4423232d7 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:39.705736 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 68791b3adccd4168b55a2da4423232d7 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-68791b3adccd4168b55a2da4423232d7
I20240503 01:59:39.706826 7392 tablet_copy_source_session.cc:205] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:39.709208 11334 tablet_replica.cc:330] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: stopping tablet replica
I20240503 01:59:39.709383 11334 raft_consensus.cc:2229] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.709604 11334 raft_consensus.cc:2258] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.710345 11340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.712402 11334 ts_tablet_manager.cc:1875] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.712972 11340 tablet_copy_client.cc:783] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:39.713198 11340 tablet_copy_client.cc:647] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:39.713392 11260 tablet_copy_client.cc:523] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.713503 7370 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.715006 9385 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.715008 7632 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.715067 11260 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
I20240503 01:59:39.715508 10076 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.715647 9255 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.715858 10076 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.716033 11169 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.660503 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.697446 11105 raft_consensus.cc:1050] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER 72577fce0313400ea862bf873738af54 to VOTER
I20240503 01:59:39.718073 10236 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.718739 11105 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.719579 9255 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.720902 9385 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.720811 7633 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.721376 10339 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.722048 7237 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.722499 10339 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.722915 10954 log.cc:1196] T cc68f31e6ef8411e96d46b2419bf9676 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/cc68f31e6ef8411e96d46b2419bf9676
I20240503 01:59:39.723917 7632 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.724426 10339 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.724381 7095 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.725078 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9796308eacd94fb5ad3cb524aa590cfa with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.725759 9384 raft_consensus.cc:2932] T 9796308eacd94fb5ad3cb524aa590cfa P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.728057 10954 tablet_replica.cc:330] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.728246 10954 raft_consensus.cc:2229] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.728502 10954 raft_consensus.cc:2258] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.729362 10954 ts_tablet_manager.cc:1875] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.731274 11334 ts_tablet_manager.cc:1888] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 01:59:39.731356 11334 log.cc:1196] T a0b1a4b2ede6493f9e4e094649b4b57d P c778431b13784c7c81f3924b7f73e664: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 01:59:39.731763 10954 ts_tablet_manager.cc:1888] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.731904 10954 log.cc:1196] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c95d3c29bfbd4899b27b770bf9d3f4a6
I20240503 01:59:39.731956 7085 catalog_manager.cc:4830] TS c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): tablet a0b1a4b2ede6493f9e4e094649b4b57d (table rebalance_test_table_9 [id=304a6c7b52ea465fb77b2bd88c2fda58]) successfully deleted
I20240503 01:59:39.732620 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet cc68f31e6ef8411e96d46b2419bf9676 (table rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]) successfully deleted
I20240503 01:59:39.732777 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet c95d3c29bfbd4899b27b770bf9d3f4a6 (table rebalance_test_table_2 [id=8d5ead3760884fe38d78dd9dcf4fc139]) successfully deleted
I20240503 01:59:39.733196 11064 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.735320 9384 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.735368 7632 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.735617 7237 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.738009 7096 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, 72577fce0313400ea862bf873738af54 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.744143 7502 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:39.745574 9384 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.746009 10335 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.750689 7237 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.751058 11105 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.753242 8993 raft_consensus.cc:1203] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.757866 11105 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.759639 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.760316 9384 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.761128 7237 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 01:59:39.761293 7092 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.762367 11304 tablet_copy_client.cc:523] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.764042 11304 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap starting.
I20240503 01:59:39.766101 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:39.766311 11102 tablet_replica.cc:330] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:39.766427 11102 raft_consensus.cc:2229] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.766598 11102 raft_consensus.cc:2258] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.767359 11102 ts_tablet_manager.cc:1875] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.769531 11102 ts_tablet_manager.cc:1888] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.770257 11102 log.cc:1196] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ebe541161c84bd6b11bcd3d13d3bd2e
I20240503 01:59:39.770846 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 8ebe541161c84bd6b11bcd3d13d3bd2e (table rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]) successfully deleted
I20240503 01:59:39.771346 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 9796308eacd94fb5ad3cb524aa590cfa with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:39.771615 11095 tablet_replica.cc:330] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.771740 11095 raft_consensus.cc:2229] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.771914 11095 raft_consensus.cc:2258] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.772580 11095 ts_tablet_manager.cc:1875] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.774546 11095 ts_tablet_manager.cc:1888] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.774613 11095 log.cc:1196] T 9796308eacd94fb5ad3cb524aa590cfa P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9796308eacd94fb5ad3cb524aa590cfa
I20240503 01:59:39.775079 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 9796308eacd94fb5ad3cb524aa590cfa (table rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]) successfully deleted
I20240503 01:59:39.779343 11340 tablet_copy_client.cc:523] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:39.780236 7632 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.780952 11340 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
I20240503 01:59:39.782232 7237 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:39.782402 7502 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:39.782652 11125 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:39.782977 7570 consensus_peers.cc:485] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ba04b45ae1343f3aa4e5a82bbf7f486. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.783334 11125 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:39.810909 7370 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
W20240503 01:59:39.812856 7306 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97ffb59d3ab64d03aca5d30a7c3be1e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.813433 9255 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.813460 7239 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.813933 11169 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.814066 10077 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 01:59:39.815752 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.815927 10077 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.816493 7239 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.820814 7090 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.821476 9255 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:39.824735 11340 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.825179 11340 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:39.825549 11340 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.008s
I20240503 01:59:39.826187 11340 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.826462 11340 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
I20240503 01:59:39.826637 11340 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:39.828274 11340 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:39.828881 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-68791b3adccd4168b55a2da4423232d7 received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:39.828972 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-68791b3adccd4168b55a2da4423232d7 on tablet 68791b3adccd4168b55a2da4423232d7 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:39.830224 11278 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.830651 11278 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:39.830935 11278 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.317s user 0.037s sys 0.000s
I20240503 01:59:39.846716 11278 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.847048 11278 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:39.847273 11278 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.850332 11278 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:39.851742 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-8ad24b47690b4263bdf4a9157db7c17f received from {username='slave'} at 127.0.99.134:53659
I20240503 01:59:39.851841 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-8ad24b47690b4263bdf4a9157db7c17f on tablet 8ad24b47690b4263bdf4a9157db7c17f with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:39.852550 7370 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.854533 9125 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.855360 7502 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.855641 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:34354
I20240503 01:59:39.855823 11317 tablet_replica.cc:330] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: stopping tablet replica
I20240503 01:59:39.855926 11317 raft_consensus.cc:2229] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Raft consensus shutting down.
I20240503 01:59:39.856007 11317 raft_consensus.cc:2258] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Raft consensus is shut down!
I20240503 01:59:39.857267 11317 ts_tablet_manager.cc:1875] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.858672 10076 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.859025 10076 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.862031 7632 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:39.862524 11169 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:39.864106 11169 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.867060 9125 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.868033 7502 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.871037 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68791b3adccd4168b55a2da4423232d7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.872159 7632 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.872689 11317 ts_tablet_manager.cc:1888] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.872820 11317 log.cc:1196] T 8ad24b47690b4263bdf4a9157db7c17f P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:39.874265 7086 catalog_manager.cc:4830] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): tablet 8ad24b47690b4263bdf4a9157db7c17f (table rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]) successfully deleted
I20240503 01:59:39.878686 7090 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e reported cstate change: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.886827 9365 tablet_service.cc:1514] Processing DeleteTablet for tablet 68791b3adccd4168b55a2da4423232d7 with delete_type TABLET_DATA_TOMBSTONED (TS 72577fce0313400ea862bf873738af54 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41514
I20240503 01:59:39.888202 11420 tablet_replica.cc:330] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: stopping tablet replica
I20240503 01:59:39.888435 11420 raft_consensus.cc:2229] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 01:59:39.888600 11420 raft_consensus.cc:2258] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 01:59:39.889909 11420 ts_tablet_manager.cc:1875] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.898625 11420 ts_tablet_manager.cc:1888] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 01:59:39.898820 11420 log.cc:1196] T 68791b3adccd4168b55a2da4423232d7 P 72577fce0313400ea862bf873738af54: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/68791b3adccd4168b55a2da4423232d7
I20240503 01:59:39.898742 7370 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.900067 10287 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.900296 7237 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.900625 9125 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.900956 9384 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.901139 10076 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.902444 7237 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.903015 7502 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:39.904247 11086 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.904560 11086 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.906646 7094 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index -1 to 4, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:39.911520 10076 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.914629 9122 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.918555 9384 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.920109 7239 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.920498 11260 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:39.923393 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d332a7c255b44589d54b1f583a2f9c4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.923851 7099 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.925042 11260 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:39.925438 7084 catalog_manager.cc:4830] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): tablet 68791b3adccd4168b55a2da4423232d7 (table rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]) successfully deleted
I20240503 01:59:39.925961 11260 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.211s user 0.036s sys 0.000s
I20240503 01:59:39.926926 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d332a7c255b44589d54b1f583a2f9c4 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:39.927213 10954 tablet_replica.cc:330] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:39.927377 10954 raft_consensus.cc:2229] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.927528 10954 raft_consensus.cc:2258] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.928480 10954 ts_tablet_manager.cc:1875] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.929665 7370 consensus_queue.cc:237] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:39.930867 10954 ts_tablet_manager.cc:1888] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:39.930938 10954 log.cc:1196] T 7d332a7c255b44589d54b1f583a2f9c4 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7d332a7c255b44589d54b1f583a2f9c4
I20240503 01:59:39.931489 9122 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.931514 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 7d332a7c255b44589d54b1f583a2f9c4 (table rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]) successfully deleted
I20240503 01:59:39.932242 11260 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:39.932350 10236 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.933079 9384 raft_consensus.cc:1261] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.933614 10236 consensus_queue.cc:1035] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.933784 11260 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:39.935128 11260 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:39.935622 11169 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.936610 11260 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 01:59:39.937067 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-97ffb59d3ab64d03aca5d30a7c3be1e1 received from {username='slave'} at 127.0.99.133:38831
I20240503 01:59:39.937134 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-97ffb59d3ab64d03aca5d30a7c3be1e1 on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:39.937484 9122 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:39.939785 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d332a7c255b44589d54b1f583a2f9c4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.940068 9384 raft_consensus.cc:2932] T 7d332a7c255b44589d54b1f583a2f9c4 P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
W20240503 01:59:39.940860 7086 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 7d332a7c255b44589d54b1f583a2f9c4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 8 but the committed config has opid_index of 9
I20240503 01:59:39.942541 7090 catalog_manager.cc:5484] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.945921 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d332a7c255b44589d54b1f583a2f9c4 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:39.946142 11095 tablet_replica.cc:330] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:39.946300 11095 raft_consensus.cc:2229] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.946473 11095 raft_consensus.cc:2258] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.947321 11095 ts_tablet_manager.cc:1875] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.949659 11095 ts_tablet_manager.cc:1888] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:39.949744 11095 log.cc:1196] T 7d332a7c255b44589d54b1f583a2f9c4 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d332a7c255b44589d54b1f583a2f9c4
I20240503 01:59:39.950577 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 7d332a7c255b44589d54b1f583a2f9c4 (table rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]) successfully deleted
I20240503 01:59:39.965312 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:39.976181 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.976444 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:39.988819 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:39.999722 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.002995 7502 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
W20240503 01:59:40.004951 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ebe541161c84bd6b11bcd3d13d3bd2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.005401 7239 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.005899 11105 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:40.006057 11304 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.008132 11105 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:40.011085 7239 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:40.012617 7092 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.013620 9384 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.014062 11057 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:40.015254 9384 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:40.015859 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:40.025650 11304 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Bootstrap complete.
I20240503 01:59:40.025925 11304 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent bootstrapping tablet: real 0.262s user 0.035s sys 0.000s
I20240503 01:59:40.030898 11304 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:40.031093 11304 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8ecb5fc8d804c9b9ca42ee4b2377f4e, State: Initialized, Role: LEARNER
I20240503 01:59:40.031247 11304 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:40.031833 11304 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:40.032630 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-ec179e5e683d4ba099c459331ea5e578 received from {username='slave'} at 127.0.99.134:45331
I20240503 01:59:40.032716 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session c8ecb5fc8d804c9b9ca42ee4b2377f4e-ec179e5e683d4ba099c459331ea5e578 on tablet ec179e5e683d4ba099c459331ea5e578 with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:40.037536 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.039779 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:40.046216 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:40.069120 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.078868 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.091737 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.110544 7370 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.111879 8993 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.112381 10236 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.113993 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.114205 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.114014 11169 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.114480 8993 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.114742 10236 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.115787 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.117116 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad24b47690b4263bdf4a9157db7c17f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.117611 7092 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 9 to 10, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.121775 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.128532 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad24b47690b4263bdf4a9157db7c17f with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:40.128870 11102 tablet_replica.cc:330] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:40.128990 11102 raft_consensus.cc:2229] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.129158 11102 raft_consensus.cc:2258] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.129882 11102 ts_tablet_manager.cc:1875] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.130205 11057 raft_consensus.cc:1050] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER c778431b13784c7c81f3924b7f73e664 to VOTER
I20240503 01:59:40.130627 11057 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:40.131758 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.132066 11102 ts_tablet_manager.cc:1888] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 01:59:40.132124 11102 log.cc:1196] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ad24b47690b4263bdf4a9157db7c17f
I20240503 01:59:40.132158 7239 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.132223 9255 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.132616 10335 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.132795 10340 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.133579 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet 8ad24b47690b4263bdf4a9157db7c17f (table rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]) successfully deleted
I20240503 01:59:40.136442 8992 raft_consensus.cc:1261] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.136478 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.136989 11057 consensus_queue.cc:1035] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.141170 11057 raft_consensus.cc:1050] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:40.141575 11057 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.142490 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.143208 8991 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.143308 7632 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.143786 11057 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.144840 10335 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.146418 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.146844 10335 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.148610 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:40.155649 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:40.183528 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.187995 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.196494 11057 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.196995 8993 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.200028 7632 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.200385 7094 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.202060 7239 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.208801 7502 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.210172 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.210237 8993 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.210639 11105 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.210978 10335 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.213016 10335 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.213615 7239 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.215186 8992 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.215377 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c18129344d124a5f8f688ec35ef69a55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.216025 7094 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.220319 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet c18129344d124a5f8f688ec35ef69a55 with delete_type TABLET_DATA_TOMBSTONED (TS efeb87c28ee04b3da5c686665b0c5c58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56156
I20240503 01:59:40.221858 11102 tablet_replica.cc:330] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: stopping tablet replica
I20240503 01:59:40.222020 11102 raft_consensus.cc:2229] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.222173 11102 raft_consensus.cc:2258] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.223644 11102 ts_tablet_manager.cc:1875] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.226228 11102 ts_tablet_manager.cc:1888] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:40.226361 11102 log.cc:1196] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c18129344d124a5f8f688ec35ef69a55
I20240503 01:59:40.226989 7084 catalog_manager.cc:4830] TS efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): tablet c18129344d124a5f8f688ec35ef69a55 (table rebalance_test_table_7 [id=1fa7c6f256e94bc9ad446cf003876203]) successfully deleted
I20240503 01:59:40.230892 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.232844 10339 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:40.237479 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.237340 7237 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:40.240128 7092 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:40.242609 8992 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:40.252799 11304 ts_tablet_manager.cc:902] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P c8ecb5fc8d804c9b9ca42ee4b2377f4e: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
F20240503 01:59:40.252987 11304 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.queue_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.queue_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_cpu_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_cpu_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_wall_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_wall_time_us) (tablet-download-pool-5ba04b45ae1343f3aa4e5a82bbf7f486.queue_time_us, tablet-download-pool-5ba04b45ae1343f3aa4e5a82bbf7f486.queue_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.queue_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.queue_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_cpu_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_cpu_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_wall_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_wall_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.queue_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.queue_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_cpu_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_cpu_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_wall_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_wall_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_cpu_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_cpu_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_wall_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_wall_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us) (tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.queue_time_us, tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.queue_time_us) (tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.run_cpu_time_us, tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.run_cpu_time_us) (tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.run_wall_time_us, tablet-download-pool-a0a526f7fced4a04b4e0327f4656cf69.run_wall_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.queue_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.queue_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_cpu_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_cpu_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_wall_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_wall_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.queue_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.queue_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_cpu_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_cpu_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_wall_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_wall_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.queue_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.queue_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_cpu_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_cpu_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_wall_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_wall_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.queue_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.queue_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_cpu_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_cpu_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_wall_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_wall_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.queue_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.queue_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_cpu_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_cpu_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_wall_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_wall_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.queue_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.queue_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_cpu_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_cpu_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_wall_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 01:59:40.253381 8993 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.259428 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.259433 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f9484480dcd google::LogMessage::Fail() at ??:0
I20240503 01:59:40.265561 7502 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:40.267238 7370 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.268252 11064 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:40.268791 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 245cba4e5728491896e7995ca0d59b36. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.271452 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:40.272449 7239 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:40.273283 10335 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f9484484b93 google::LogMessage::SendToLog() at ??:0
I20240503 01:59:40.286866 10335 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.290181 7239 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.293705 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f94844807cc google::LogMessage::Flush() at ??:0
I20240503 01:59:40.295821 7092 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.296039 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.296811 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.301699 9255 raft_consensus.cc:2932] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c778431b13784c7c81f3924b7f73e664 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 01:59:40.309602 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f9484481f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 01:59:40.318229 7370 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.326009 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:40.331571 9105 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.355329 11532 ts_tablet_manager.cc:902] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:40.355813 11532 tablet_copy_client.cc:308] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:40.356493 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer 72577fce0313400ea862bf873738af54 ({username='slave'} at 127.0.99.136:54521)
I20240503 01:59:40.356591 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 from peer 72577fce0313400ea862bf873738af54 at {username='slave'} at 127.0.99.136:54521: session id = 72577fce0313400ea862bf873738af54-97ffb59d3ab64d03aca5d30a7c3be1e1
I20240503 01:59:40.357463 7392 tablet_copy_source_session.cc:205] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:40.358390 11532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.361554 11532 tablet_copy_client.cc:783] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:40.361802 11532 tablet_copy_client.cc:647] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:40.363459 7370 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:40.364888 7502 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.364912 9122 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.365284 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.365648 11168 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.365911 7632 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.366884 11168 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.368590 11168 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.369535 7502 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.371623 7094 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.372083 7632 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.374879 7370 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 01:59:40.375353 9125 raft_consensus.cc:1203] T ec179e5e683d4ba099c459331ea5e578 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:40.375473 9122 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
@ 0x7f94858ea044 kudu::TraceMetrics::InternName() at ??:0
I20240503 01:59:40.377152 7632 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.377313 9125 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.377553 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f94858cef98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 01:59:40.379492 10236 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.381217 11086 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.381630 7632 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.381644 9125 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 01:59:40.382802 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68791b3adccd4168b55a2da4423232d7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.383705 7094 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:40.387624 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 68791b3adccd4168b55a2da4423232d7 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:40.387841 10954 tablet_replica.cc:330] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
I20240503 01:59:40.388016 10954 raft_consensus.cc:2229] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.388224 10954 raft_consensus.cc:2258] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.388769 7370 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:40.390717 7632 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 01:59:40.391054 7306 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.391229 9522 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.393213 10954 ts_tablet_manager.cc:1875] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.393477 9522 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.395718 7632 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.396692 9125 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
@ 0x7f94858ccefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 01:59:40.397132 11086 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.397672 10954 ts_tablet_manager.cc:1888] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:40.397761 10954 log.cc:1196] T 68791b3adccd4168b55a2da4423232d7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/68791b3adccd4168b55a2da4423232d7
I20240503 01:59:40.398782 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 68791b3adccd4168b55a2da4423232d7 (table rebalance_test_table_0 [id=37323b4f15ad49a1b2a00e2eb771c02f]) successfully deleted
I20240503 01:59:40.398720 9125 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.399080 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 68791b3adccd4168b55a2da4423232d7 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:40.400214 7091 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.406646 11532 tablet_copy_client.cc:523] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.408309 11532 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Bootstrap starting.
@ 0x7f948847f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 01:59:40.414052 7632 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:40.416155 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:40.416734 9255 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
@ 0x7f9488488a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 01:59:40.417796 8993 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 01:59:40.418812 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.419128 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95d3c29bfbd4899b27b770bf9d3f4a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.419229 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.419483 9122 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c8ecb5fc8d804c9b9ca42ee4b2377f4e [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.419651 10076 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f948850976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 01:59:40.424202 11260 ts_tablet_manager.cc:902] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:40.424703 11260 tablet_copy_client.cc:308] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:40.425225 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer e648eca559ed4e5ab24a1b98755aa3eb ({username='slave'} at 127.0.99.133:55451)
I20240503 01:59:40.425328 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer e648eca559ed4e5ab24a1b98755aa3eb at {username='slave'} at 127.0.99.133:55451: session id = e648eca559ed4e5ab24a1b98755aa3eb-8ebe541161c84bd6b11bcd3d13d3bd2e
@ 0x7f948851d4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 01:59:40.426265 7523 tablet_copy_source_session.cc:205] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f9488508226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 01:59:40.429068 10076 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.430116 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f9488517c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55adeab0875c std::function<>::operator()() at ??:0
I20240503 01:59:40.434739 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f94858d1e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 01:59:40.436975 10287 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.437206 7632 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:40.438158 11532 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.438482 11532 tablet_bootstrap.cc:492] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Bootstrap complete.
I20240503 01:59:40.438697 11532 ts_tablet_manager.cc:1367] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240503 01:59:40.438830 9254 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.439186 11532 raft_consensus.cc:348] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:40.439313 11532 raft_consensus.cc:729] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72577fce0313400ea862bf873738af54, State: Initialized, Role: LEARNER
@ 0x7f94858d26ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 01:59:40.439445 11532 consensus_queue.cc:260] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:40.439749 11532 ts_tablet_manager.cc:1398] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 72577fce0313400ea862bf873738af54: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:40.440011 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.440248 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 72577fce0313400ea862bf873738af54-97ffb59d3ab64d03aca5d30a7c3be1e1 received from {username='slave'} at 127.0.99.136:54521
I20240503 01:59:40.440316 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 72577fce0313400ea862bf873738af54-97ffb59d3ab64d03aca5d30a7c3be1e1 on tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with peer 72577fce0313400ea862bf873738af54
I20240503 01:59:40.440882 11260 tablet_copy_client.cc:783] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:40.441051 11260 tablet_copy_client.cc:647] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:40.441179 11125 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:40.441587 7570 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.441540 10076 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.441679 11125 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f94858d403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 01:59:40.442867 11125 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
@ 0x55adeab0875c std::function<>::operator()() at ??:0
I20240503 01:59:40.444723 7632 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.445183 9255 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.445317 8459 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
@ 0x7f94858c1807 kudu::Thread::SuperviseThread() at ??:0
I20240503 01:59:40.447693 7092 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.448570 11260 tablet_copy_client.cc:523] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:40.449154 7091 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7f94860ff6db start_thread at ??:0
I20240503 01:59:40.449879 9255 raft_consensus.cc:2932] T c95d3c29bfbd4899b27b770bf9d3f4a6 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:40.450336 11260 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap starting.
@ 0x7f948363771f clone at ??:0
I20240503 01:59:40.455796 7502 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.456358 7091 catalog_manager.cc:5484] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.471923 8992 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.493184 7239 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.494693 8992 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.494863 7632 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.495231 11421 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.495419 10337 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.505730 7370 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.507872 7502 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.507901 8993 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.508471 11168 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.508805 11168 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.511466 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:40.518162 11587 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.518556 7632 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.512862 11168 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:40.520771 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:40.520856 7370 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 01:59:40.524531 9384 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 01:59:40.525040 10076 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:40.525089 9522 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.525470 8992 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 01:59:40.525847 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:40.527499 7502 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.528137 7094 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.531116 9255 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:40.537079 7239 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.530872 7098 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.532828 9384 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:40.554378 7632 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.528270 8993 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.531148 8991 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.535986 8992 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:40.544888 11260 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:40.556727 11260 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Bootstrap complete.
I20240503 01:59:40.556907 11260 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Time spent bootstrapping tablet: real 0.107s user 0.031s sys 0.000s
I20240503 01:59:40.532151 10076 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:40.557400 11260 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:40.557508 11260 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e648eca559ed4e5ab24a1b98755aa3eb, State: Initialized, Role: LEARNER
I20240503 01:59:40.557601 11260 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:40.557880 11260 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 01:59:40.533499 7306 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.558449 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-8ebe541161c84bd6b11bcd3d13d3bd2e received from {username='slave'} at 127.0.99.133:55451
I20240503 01:59:40.559477 7091 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
W20240503 01:59:40.538390 7307 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.608206 7307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:42469, user_credentials={real_user=slave}} blocked reactor thread for 69833.4us
I20240503 01:59:40.609133 11168 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:40.609897 7370 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
W20240503 01:59:40.612291 7306 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d00b8fcb29f47298643903ec0cba606. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.617789 11168 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 01:59:40.627751 7092 catalog_manager.cc:5484] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 11 to 12, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.558542 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session e648eca559ed4e5ab24a1b98755aa3eb-8ebe541161c84bd6b11bcd3d13d3bd2e on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with peer e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:40.612713 7502 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.685478 9365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.691380 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:40.625777 7370 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.657757 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:40.674412 11169 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:40.560271 7176 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81086ff737c84d9d9f04743d31c6151b. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.713413 7176 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:43981, user_credentials={real_user=slave}} blocked reactor thread for 153155us
I20240503 01:59:40.562633 11623 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.577126 7239 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b"
dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55300
I20240503 01:59:40.714584 7239 raft_consensus.cc:595] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:40.614595 7237 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.716406 7240 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.717819 9384 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 01:59:40.719076 7221 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.720670 10076 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 01:59:40.721226 7308 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6649e1feab5b40738003119a9122f07a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.728451 11532 ts_tablet_manager.cc:902] T 245cba4e5728491896e7995ca0d59b36 P 72577fce0313400ea862bf873738af54: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
F20240503 01:59:40.738474 11532 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-245cba4e5728491896e7995ca0d59b36.queue_time_us, tablet-download-pool-245cba4e5728491896e7995ca0d59b36.queue_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.queue_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.queue_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_cpu_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_cpu_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_wall_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_wall_time_us) (tablet-download-pool-40260acad62242c3a72197d2dafae187.queue_time_us, tablet-download-pool-40260acad62242c3a72197d2dafae187.queue_time_us) (tablet-download-pool-40260acad62242c3a72197d2dafae187.run_cpu_time_us, tablet-download-pool-40260acad62242c3a72197d2dafae187.run_cpu_time_us) (tablet-download-pool-40260acad62242c3a72197d2dafae187.run_wall_time_us, tablet-download-pool-40260acad62242c3a72197d2dafae187.run_wall_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.queue_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.queue_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_cpu_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_cpu_time_us) (tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_wall_time_us, tablet-download-pool-68791b3adccd4168b55a2da4423232d7.run_wall_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.queue_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.queue_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_cpu_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_cpu_time_us) (tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_wall_time_us, tablet-download-pool-7d332a7c255b44589d54b1f583a2f9c4.run_wall_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.queue_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.queue_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_cpu_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_cpu_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_wall_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_wall_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.queue_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.queue_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_cpu_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_cpu_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_wall_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_wall_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.queue_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.queue_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_cpu_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_cpu_time_us) (tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_wall_time_us, tablet-download-pool-b4628afbfeff4f8bbaba5da1e89e4850.run_wall_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.queue_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.queue_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_cpu_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_cpu_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_wall_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_wall_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.queue_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.queue_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_cpu_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_cpu_time_us) (tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_wall_time_us, tablet-download-pool-ec179e5e683d4ba099c459331ea5e578.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f1343ef1dcd google::LogMessage::Fail() at ??:0
I20240503 01:59:40.724026 11421 raft_consensus.cc:979] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2: : Instructing follower efeb87c28ee04b3da5c686665b0c5c58 to start an election
I20240503 01:59:40.730840 9384 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 01:59:40.743779 11421 raft_consensus.cc:1067] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Signalling peer efeb87c28ee04b3da5c686665b0c5c58 to start an election
I20240503 01:59:40.630453 7632 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:40.619930 9255 raft_consensus.cc:2932] T a432cd5fe20941d6bc73dae386d85170 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
@ 0x7f1343ef5b93 google::LogMessage::SendToLog() at ??:0
I20240503 01:59:40.561378 8990 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.721509 11169 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:40.725677 7307 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.730232 10076 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 01:59:40.729866 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 01:59:40.748988 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.749140 7632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b"
dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
from {username='slave'} at 127.0.99.129:59655
I20240503 01:59:40.749274 7632 raft_consensus.cc:482] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:40.749325 7632 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:40.612735 8993 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.750262 7632 raft_consensus.cc:504] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.723487 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 01:59:40.750732 7632 leader_election.cc:288] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: Requested vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145)
I20240503 01:59:40.729492 8992 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:40.751338 7239 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.751852 7092 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 72577fce0313400ea862bf873738af54 reported cstate change: config changed from index 13 to 14, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 01:59:40.753093 9522 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:40.753405 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:40.753643 9522 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:40.754006 11168 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:40.754578 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.754869 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:40.754981 7239 raft_consensus.cc:3032] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:40.755020 7239 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Running, Role: LEADER
I20240503 01:59:40.755025 11558 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.755204 7239 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.755404 7239 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:40.755537 7613 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:40.756448 7239 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 2.
I20240503 01:59:40.757449 11558 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 01:59:40.757992 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 01:59:40.760378 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.761019 7092 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.766625 11169 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:40.768064 11169 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
@ 0x7f1343ef17cc google::LogMessage::Flush() at ??:0
I20240503 01:59:40.770078 7239 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
@ 0x7f1343ef2f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f134535b044 kudu::TraceMetrics::InternName() at ??:0
I20240503 01:59:40.775399 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.776105 7632 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
@ 0x7f134533ff98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 01:59:40.777168 8992 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:40.778643 8991 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
@ 0x7f134533defc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 01:59:40.781558 7092 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 10 to 11, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:40.782636 7502 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:40.785305 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.785547 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d00b8fcb29f47298643903ec0cba606 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
@ 0x7f1347ef077a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 01:59:40.787802 7240 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
@ 0x7f1347ef9a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 01:59:40.790975 7092 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.795555 7370 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
@ 0x7f1347f7a76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240503 01:59:40.799432 7306 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d00b8fcb29f47298643903ec0cba606. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.801396 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.801731 10076 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7f1347f8e4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 01:59:40.803042 8992 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:40.803392 10076 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:40.804399 10076 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
@ 0x7f1347f79226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 01:59:40.806182 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d00b8fcb29f47298643903ec0cba606 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:40.807492 7239 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.807595 7094 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7f1347f88c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 01:59:40.810730 7483 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d00b8fcb29f47298643903ec0cba606 with delete_type TABLET_DATA_TOMBSTONED (TS 5b46f0d4b665497b8d7e57bf4d5f36cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57224
I20240503 01:59:40.810967 10954 tablet_replica.cc:330] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: stopping tablet replica
@ 0x55f62f66875c std::function<>::operator()() at ??:0
I20240503 01:59:40.811108 10954 raft_consensus.cc:2229] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.811372 10954 raft_consensus.cc:2258] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.812294 10954 ts_tablet_manager.cc:1875] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f1345342e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 01:59:40.815112 10954 ts_tablet_manager.cc:1888] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 01:59:40.815189 10954 log.cc:1196] T 0d00b8fcb29f47298643903ec0cba606 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d00b8fcb29f47298643903ec0cba606
I20240503 01:59:40.815798 7086 catalog_manager.cc:4830] TS 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): tablet 0d00b8fcb29f47298643903ec0cba606 (table rebalance_test_table_4 [id=824ac75858614f5a9292bf5cfabf8c80]) successfully deleted
@ 0x7f13453436ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 01:59:40.817463 8992 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
W20240503 01:59:40.819469 7177 connection.cc:535] client connection to 127.0.99.134:43615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f134534503b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
W20240503 01:59:40.819681 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:40.820432 7570 connection.cc:535] client connection to 127.0.99.134:43615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:40.820554 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x55f62f66875c std::function<>::operator()() at ??:0
W20240503 01:59:40.821633 7308 connection.cc:535] client connection to 127.0.99.134:43615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:40.821743 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f1345332807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1345b706db start_thread at ??:0
@ 0x7f13430a871f clone at ??:0
W20240503 01:59:40.831835 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.831923 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.839571 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81086ff737c84d9d9f04743d31c6151b" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
I20240503 01:59:40.839787 8992 raft_consensus.cc:3037] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:40.841043 7502 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.842772 7632 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:40.843147 11105 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 01:59:40.843254 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:40.843307 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.843356 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:40.843429 7570 leader_election.cc:334] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:40.843492 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.846027 8992 raft_consensus.cc:2454] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 2.
I20240503 01:59:40.846359 7567 leader_election.cc:302] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, e648eca559ed4e5ab24a1b98755aa3eb, efeb87c28ee04b3da5c686665b0c5c58; no voters: c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:40.847704 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527af716906b42729952db28e790bafd. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.847761 11557 raft_consensus.cc:2781] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:40.847846 11557 raft_consensus.cc:686] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:40.848008 11557 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:40.850446 7091 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 1 to 2, leader changed from 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), now has a pending config: VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133), NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:40.856034 7633 consensus_queue.cc:237] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:40.860517 9255 raft_consensus.cc:1261] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.860906 11125 consensus_queue.cc:1035] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:40.865993 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.867317 11105 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:40.867981 11557 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.869762 7633 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
W20240503 01:59:40.870213 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.870467 7091 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 11 to 12, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.873704 9255 raft_consensus.cc:2932] T be1f4d2622c6452b869f88a301a5e9bc P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:40.875073 8992 raft_consensus.cc:1203] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:40.875991 7094 catalog_manager.cc:5484] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 9 to 10, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.880913 10236 raft_consensus.cc:1050] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:40.892060 10236 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:40.893936 8992 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.894397 11086 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.895493 7239 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.895928 9255 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:40.895934 11169 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.896479 11086 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.898615 11168 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.900580 8992 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.901055 7239 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.902873 7092 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:40.908581 7370 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:40.910532 8992 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.911520 11169 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:40.911623 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:40.914279 7239 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.914638 9254 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:40.914800 11169 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:40.916210 10236 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:40.916553 7502 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 01:59:40.917361 11777 ts_tablet_manager.cc:902] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:40.918124 11777 tablet_copy_client.cc:308] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:40.918669 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a432cd5fe20941d6bc73dae386d85170 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:51673)
I20240503 01:59:40.918758 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a432cd5fe20941d6bc73dae386d85170 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:51673: session id = efeb87c28ee04b3da5c686665b0c5c58-a432cd5fe20941d6bc73dae386d85170
I20240503 01:59:40.918781 7240 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:40.918902 9440 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8Tfalf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 01:59:35.414249 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.414498 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.419081 9445 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:36.424686 9445 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:37.049820 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.049921 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.049968 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.050050 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.050119 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.051425 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.061287 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.061385 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.061456 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.061566 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.062417 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.063174 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.063254 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.063836 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.064102 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:37.151249 9445 rebalancer_tool.cc:1074] tablet 27c539b143674bdd8e3e5aaa4b78c1e9: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.216634 9445 rebalancer_tool.cc:1074] tablet b4628afbfeff4f8bbaba5da1e89e4850: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.287235 9445 rebalancer_tool.cc:1074] tablet a432cd5fe20941d6bc73dae386d85170: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.353358 9445 rebalancer_tool.cc:1074] tablet 9796308eacd94fb5ad3cb524aa590cfa: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.397926 9445 rebalancer_tool.cc:1074] tablet 3b15d48b2fa04269b98c6f801f54df4f: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.459966 9445 rebalancer_tool.cc:1074] tablet a0b1a4b2ede6493f9e4e094649b4b57d: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.478322 9445 rebalancer_tool.cc:1084] tablet 5f35ccb9c21b4c10b9187596d2f6929c: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c778431b13784c7c81f3924b7f73e664' move ignored: Illegal state: tablet 5f35ccb9c21b4c10b9187596d2f6929c: replica c778431b13784c7c81f3924b7f73e664 is already present
I20240503 01:59:37.478420 9445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:38.491266 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.491384 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:38.491473 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:38.491499 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:38.491561 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.491593 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:38.491619 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:38.491647 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:38.491675 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
I20240503 01:59:38.758971 9445 rebalancer_tool.cc:1074] tablet 527af716906b42729952db28e790bafd: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:38.976420 9445 rebalancer_tool.cc:1074] tablet 8ad24b47690b4263bdf4a9157db7c17f: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:39.229688 9445 rebalancer_tool.cc:1074] tablet 68791b3adccd4168b55a2da4423232d7: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:39.505656 9445 rebalancer_tool.cc:1084] tablet c18129344d124a5f8f688ec35ef69a55: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move ignored: Illegal state: tablet c18129344d124a5f8f688ec35ef69a55: replica e648eca559ed4e5ab24a1b98755aa3eb is already present
I20240503 01:59:39.505738 9445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:40.229609 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server 5b46f0d4b665497b8d7e57bf4d5f36cb
W20240503 01:59:40.231642 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:40.231812 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:40.231966 9445 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:40.474279 9445 rebalancer_tool.cc:1074] tablet f0539dded7554723b9e918e8cdfbab20: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:40.560465 9445 rebalancer_tool.cc:1074] tablet 6649e1feab5b40738003119a9122f07a: 'e648eca559ed4e5ab24a1b98755aa3eb' -> '4356d1cb1050473a8d9e50ff09a24dc2' move scheduled
I20240503 01:59:40.573735 9445 rebalancer_tool.cc:1084] tablet c18129344d124a5f8f688ec35ef69a55: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c778431b13784c7c81f3924b7f73e664' move ignored: Not found: Peer with uuid efeb87c28ee04b3da5c686665b0c5c58 not found in consensus config
I20240503 01:59:40.573822 9445 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:40.822377 11696 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:40.681297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:43615 (local address 127.0.0.1:34396)
0503 01:59:40.687066 (+ 5769us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:40.687080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:40.687168 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:40.821556 (+134388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5711}
Illegal state: tablet server c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): unacceptable health status UNAVAILABLE
I20240503 01:59:40.922150 9254 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 01:59:40.989595 9254 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.992640 7092 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.996876 11819 ts_tablet_manager.cc:902] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:40.998731 11819 tablet_copy_client.cc:308] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:41.008961 7221 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.021240 8973 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:41.033728 9236 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.920097 11064 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:40.919793 7392 tablet_copy_source_session.cc:205] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.019670 7483 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:40.922890 11169 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.927269 7370 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:41.064296 8992 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:40.973877 7371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7"
dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42782
I20240503 01:59:41.064890 7371 raft_consensus.cc:595] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.065274 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.072047 9254 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 01:59:41.066581 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): Couldn't send request to peer bed387c0ff9241688cdf45f364dc6f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40260acad62242c3a72197d2dafae187. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.100157 7439 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.130:46181, user_credentials={real_user=slave}} blocked reactor thread for 178632us
W20240503 01:59:40.920676 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.044847 7613 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 01:59:41.100944 7570 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:43615, user_credentials={real_user=slave}} blocked reactor thread for 180277us
I20240503 01:59:41.101979 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:59655)
W20240503 01:59:40.990489 7308 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.102070 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:59655: session id = 4356d1cb1050473a8d9e50ff09a24dc2-c95d3c29bfbd4899b27b770bf9d3f4a6
W20240503 01:59:41.102079 7308 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:43615, user_credentials={real_user=slave}} blocked reactor thread for 111599us
I20240503 01:59:41.102286 11777 ts_tablet_manager.cc:1875] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 01:59:41.065955 8991 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:41.065594 7240 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.102952 7653 tablet_copy_source_session.cc:205] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.070133 11064 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 01:59:41.104585 11777 ts_tablet_manager.cc:1888] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 01:59:41.104738 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a432cd5fe20941d6bc73dae386d85170. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:41.066881 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.105901 11777 tablet_copy_client.cc:783] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.106053 11777 tablet_copy_client.cc:647] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.106559 11168 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:41.071163 7239 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 01:59:41.108682 11168 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:41.109126 9254 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:41.109978 7371 consensus_queue.cc:237] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:41.111003 9254 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:41.112099 7092 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 01:59:41.113034 10236 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:41.113569 7093 catalog_manager.cc:5484] T 5f35ccb9c21b4c10b9187596d2f6929c P c778431b13784c7c81f3924b7f73e664 reported cstate change: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:41.114850 8992 raft_consensus.cc:1261] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:41.116042 9522 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.117431 9522 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:41.119488 9254 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.120332 10076 raft_consensus.cc:979] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a: : Instructing follower efeb87c28ee04b3da5c686665b0c5c58 to start an election
I20240503 01:59:41.120399 10076 raft_consensus.cc:1067] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Signalling peer efeb87c28ee04b3da5c686665b0c5c58 to start an election
I20240503 01:59:41.120952 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:41.122220 11169 consensus_queue.cc:1035] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:41.122545 7093 catalog_manager.cc:5484] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.124060 11874 ts_tablet_manager.cc:902] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:41.124644 8991 raft_consensus.cc:2932] T 5f35ccb9c21b4c10b9187596d2f6929c P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 01:59:41.124841 11874 tablet_copy_client.cc:308] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:41.127880 7633 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7"
dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
from {username='slave'} at 127.0.99.130:59195
I20240503 01:59:41.128012 7633 raft_consensus.cc:482] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:41.128065 7633 raft_consensus.cc:3037] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.128872 7633 raft_consensus.cc:504] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:41.129137 7633 leader_election.cc:288] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: Requested vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615)
W20240503 01:59:41.132208 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.132334 8992 raft_consensus.cc:2932] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.133200 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 6649e1feab5b40738003119a9122f07a from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:51591)
I20240503 01:59:41.133287 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 6649e1feab5b40738003119a9122f07a from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:51591: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-6649e1feab5b40738003119a9122f07a
I20240503 01:59:41.134222 7392 tablet_copy_source_session.cc:205] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.136906 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:47696
I20240503 01:59:41.137122 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68791b3adccd4168b55a2da4423232d7" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:41.137202 11885 tablet_replica.cc:330] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 01:59:41.137207 7350 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:41.137231 7371 raft_consensus.cc:3032] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:41.137282 7371 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:41.137404 7371 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:41.137553 7371 raft_consensus.cc:3037] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.137641 11819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95d3c29bfbd4899b27b770bf9d3f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.138183 11885 raft_consensus.cc:2229] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.138432 7371 raft_consensus.cc:2454] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb87c28ee04b3da5c686665b0c5c58 in term 2.
W20240503 01:59:41.140264 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.140551 7570 leader_election.cc:302] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:41.140892 11125 raft_consensus.cc:2781] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:41.140977 11125 raft_consensus.cc:686] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Becoming Leader. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Running, Role: LEADER
I20240503 01:59:41.141132 11125 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 01:59:41.144274 7090 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: term changed from 1 to 2, leader changed from bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) to efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.145402 11777 tablet_copy_client.cc:523] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.145481 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:41.145961 7306 connection.cc:535] client connection to 127.0.99.136:33955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.146003 7084 connection.cc:535] client connection to 127.0.99.136:33955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.146102 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.146210 7437 connection.cc:535] client connection to 127.0.99.136:33955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.146571 7568 connection.cc:535] client connection to 127.0.99.136:33955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 01:59:41.147114 11819 tablet_copy_client.cc:783] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.147178 11777 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 01:59:41.147302 11819 tablet_copy_client.cc:647] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
W20240503 01:59:41.148650 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.149216 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.151629 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.152283 7570 leader_election.cc:334] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.153458 11819 tablet_copy_client.cc:523] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.155207 11819 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
W20240503 01:59:41.157663 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.157778 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.158332 7437 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:41.158396 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.158437 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.158473 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.158710 11890 ts_tablet_manager.cc:902] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:41.159189 11890 tablet_copy_client.cc:308] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:41.159662 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 6649e1feab5b40738003119a9122f07a from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:50331)
I20240503 01:59:41.159750 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 6649e1feab5b40738003119a9122f07a from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:50331: session id = 4356d1cb1050473a8d9e50ff09a24dc2-6649e1feab5b40738003119a9122f07a
I20240503 01:59:41.160717 7392 tablet_copy_source_session.cc:205] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.160899 11874 tablet_copy_client.cc:783] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.161319 11874 tablet_copy_client.cc:647] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.161439 11890 ts_tablet_manager.cc:1875] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 01:59:41.162886 11874 tablet_copy_client.cc:523] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.164711 11890 ts_tablet_manager.cc:1888] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240503 01:59:41.164841 11874 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:41.168721 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6649e1feab5b40738003119a9122f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.170398 11890 tablet_copy_client.cc:783] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.170591 11890 tablet_copy_client.cc:647] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
W20240503 01:59:41.174050 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:41.138470 11885 raft_consensus.cc:2258] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.180824 11885 ts_tablet_manager.cc:1875] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.183210 11885 ts_tablet_manager.cc:1888] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 01:59:41.183301 11885 log.cc:1196] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/97ffb59d3ab64d03aca5d30a7c3be1e1
I20240503 01:59:41.184317 11903 ts_tablet_manager.cc:902] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 01:59:41.184895 11903 tablet_copy_client.cc:308] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 01:59:41.185367 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:49207)
I20240503 01:59:41.185469 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet b4628afbfeff4f8bbaba5da1e89e4850 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:49207: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-b4628afbfeff4f8bbaba5da1e89e4850
I20240503 01:59:41.186496 7653 tablet_copy_source_session.cc:205] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.187844 11890 tablet_copy_client.cc:523] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.189158 11890 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:41.191803 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 97ffb59d3ab64d03aca5d30a7c3be1e1 (table rebalance_test_table_1 [id=3ea4dc3283bd46ffb0691a36b400dbb1]) successfully deleted
W20240503 01:59:41.194188 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:41.204396 11903 ts_tablet_manager.cc:1875] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 01:59:41.208375 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.209046 11874 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.209369 11874 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 01:59:41.209532 11874 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.045s user 0.032s sys 0.000s
I20240503 01:59:41.209959 11874 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.210119 11874 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 01:59:41.210215 11874 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.211874 11903 ts_tablet_manager.cc:1888] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 01:59:41.212101 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4628afbfeff4f8bbaba5da1e89e4850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.214872 11819 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.215256 11819 tablet_bootstrap.cc:492] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 01:59:41.215476 11819 ts_tablet_manager.cc:1367] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.060s user 0.029s sys 0.001s
I20240503 01:59:41.216300 11903 tablet_copy_client.cc:783] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.216804 11819 raft_consensus.cc:348] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:41.216984 11903 tablet_copy_client.cc:647] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.217002 11819 raft_consensus.cc:729] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 01:59:41.217175 11819 consensus_queue.cc:260] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:41.218080 11819 ts_tablet_manager.cc:1398] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:41.218590 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-c95d3c29bfbd4899b27b770bf9d3f4a6 received from {username='slave'} at 127.0.99.129:59655
I20240503 01:59:41.218667 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-c95d3c29bfbd4899b27b770bf9d3f4a6 on tablet c95d3c29bfbd4899b27b770bf9d3f4a6 with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 01:59:41.221983 11903 tablet_copy_client.cc:523] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.222112 9437 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8Tfalf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 01:59:35.456650 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.456977 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.462603 9443 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:36.551672 9443 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:37.012014 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.013545 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.023389 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.023451 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.023553 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.023662 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.023882 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.023950 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.024008 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.024047 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:37.024082 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.024123 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.024190 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.024240 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.024276 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:37.024314 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:37.024384 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:37.024461 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
I20240503 01:59:37.097793 9443 rebalancer_tool.cc:1074] tablet b4628afbfeff4f8bbaba5da1e89e4850: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.163043 9443 rebalancer_tool.cc:1074] tablet 18468345c7924d4f88366524b56e8ab3: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.218204 9443 rebalancer_tool.cc:1074] tablet 6649e1feab5b40738003119a9122f07a: 'efeb87c28ee04b3da5c686665b0c5c58' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.250790 9443 rebalancer_tool.cc:1074] tablet a432cd5fe20941d6bc73dae386d85170: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.280377 9443 rebalancer_tool.cc:1074] tablet a0b1a4b2ede6493f9e4e094649b4b57d: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.310062 9443 rebalancer_tool.cc:1074] tablet 0d00b8fcb29f47298643903ec0cba606: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.376065 9443 rebalancer_tool.cc:1074] tablet 40260acad62242c3a72197d2dafae187: 'efeb87c28ee04b3da5c686665b0c5c58' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.407127 9443 rebalancer_tool.cc:1074] tablet 5f35ccb9c21b4c10b9187596d2f6929c: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.434669 9443 rebalancer_tool.cc:1074] tablet a0a526f7fced4a04b4e0327f4656cf69: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.512702 9443 rebalancer_tool.cc:1074] tablet fb00fcf563b74243b7c955ddacf1d6cf: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.599414 9443 rebalancer_tool.cc:1074] tablet 8ebe541161c84bd6b11bcd3d13d3bd2e: 'efeb87c28ee04b3da5c686665b0c5c58' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.637990 9443 rebalancer_tool.cc:1074] tablet 7d332a7c255b44589d54b1f583a2f9c4: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.762784 9443 rebalancer_tool.cc:1074] tablet be1f4d2622c6452b869f88a301a5e9bc: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:38.166785 9443 rebalancer_tool.cc:1084] tablet 3b15d48b2fa04269b98c6f801f54df4f: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move ignored: Illegal state: tablet 3b15d48b2fa04269b98c6f801f54df4f: replica 72577fce0313400ea862bf873738af54 is already present
I20240503 01:59:38.166893 9443 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:39.764653 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:39.764762 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 72577fce0313400ea862bf873738af54 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:39.764823 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server efeb87c28ee04b3da5c686665b0c5c58
W20240503 01:59:39.764894 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:39.764941 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:39.764986 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:39.765028 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:39.765094 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:39.765146 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:39.818284 9443 rebalancer_tool.cc:1074] tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:39.831472 9443 rebalancer_tool.cc:1084] tablet 5ba04b45ae1343f3aa4e5a82bbf7f486: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c778431b13784c7c81f3924b7f73e664' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:39.831575 9443 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:40.479135 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 5b46f0d4b665497b8d7e57bf4d5f36cb
W20240503 01:59:40.479223 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server bed387c0ff9241688cdf45f364dc6f5a
W20240503 01:59:40.479260 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server efeb87c28ee04b3da5c686665b0c5c58
W20240503 01:59:40.479358 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server e648eca559ed4e5ab24a1b98755aa3eb to server 4356d1cb1050473a8d9e50ff09a24dc2
W20240503 01:59:40.479516 9443 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
I20240503 01:59:40.523973 9443 rebalancer_tool.cc:1074] tablet 81086ff737c84d9d9f04743d31c6151b: 'efeb87c28ee04b3da5c686665b0c5c58' -> '' move scheduled
I20240503 01:59:40.785756 9443 rebalancer_tool.cc:1074] tablet 71f12a25a1db4650a6c37b5000a668c7: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:40.863380 9443 rebalancer_tool.cc:1084] tablet f0539dded7554723b9e918e8cdfbab20: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move ignored: Illegal state: tablet f0539dded7554723b9e918e8cdfbab20: replica c8ecb5fc8d804c9b9ca42ee4b2377f4e is already present
I20240503 01:59:40.863457 9443 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:41.146977 11837 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:41.002044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:33955 (local address 127.0.0.1:41576)
0503 01:59:41.002264 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:41.002278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:41.002358 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:41.146215 (+143857us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
Illegal state: tablet server c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): unacceptable health status UNAVAILABLE
I20240503 01:59:41.222921 11874 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:41.223567 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-6649e1feab5b40738003119a9122f07a received from {username='slave'} at 127.0.99.131:51591
I20240503 01:59:41.223649 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-6649e1feab5b40738003119a9122f07a on tablet 6649e1feab5b40738003119a9122f07a with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 01:59:41.225741 11903 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 01:59:41.242940 11777 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.243445 11777 tablet_bootstrap.cc:492] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 01:59:41.243726 11777 ts_tablet_manager.cc:1367] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.097s user 0.028s sys 0.004s
I20240503 01:59:41.244168 11890 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.244386 11777 raft_consensus.cc:348] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 01:59:41.244591 11777 raft_consensus.cc:729] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 01:59:41.244735 11777 consensus_queue.cc:260] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
W20240503 01:59:41.244832 7306 consensus_peers.cc:485] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.245644 11890 tablet_bootstrap.cc:492] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 01:59:41.245649 11777 ts_tablet_manager.cc:1398] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:41.245870 11890 ts_tablet_manager.cc:1367] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.057s user 0.035s sys 0.000s
I20240503 01:59:41.246172 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-a432cd5fe20941d6bc73dae386d85170 received from {username='slave'} at 127.0.99.132:51673
I20240503 01:59:41.246239 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-a432cd5fe20941d6bc73dae386d85170 on tablet a432cd5fe20941d6bc73dae386d85170 with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 01:59:41.246520 11890 raft_consensus.cc:348] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.246652 11890 raft_consensus.cc:729] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 01:59:41.246759 11890 consensus_queue.cc:260] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.247095 11890 ts_tablet_manager.cc:1398] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:41.247761 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-6649e1feab5b40738003119a9122f07a received from {username='slave'} at 127.0.99.129:50331
I20240503 01:59:41.247824 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-6649e1feab5b40738003119a9122f07a on tablet 6649e1feab5b40738003119a9122f07a with peer 4356d1cb1050473a8d9e50ff09a24dc2
W20240503 01:59:41.251794 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.262884 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81086ff737c84d9d9f04743d31c6151b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.273102 11903 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.273411 11903 tablet_bootstrap.cc:492] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 01:59:41.273658 11903 ts_tablet_manager.cc:1367] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.048s user 0.025s sys 0.005s
I20240503 01:59:41.274204 11903 raft_consensus.cc:348] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.274403 11903 raft_consensus.cc:729] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 01:59:41.274518 11903 consensus_queue.cc:260] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 01:59:41.274888 11903 ts_tablet_manager.cc:1398] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.275364 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-b4628afbfeff4f8bbaba5da1e89e4850 received from {username='slave'} at 127.0.99.131:49207
I20240503 01:59:41.275436 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-b4628afbfeff4f8bbaba5da1e89e4850 on tablet b4628afbfeff4f8bbaba5da1e89e4850 with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 01:59:41.275497 7370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa"
dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42856
I20240503 01:59:41.275643 7370 raft_consensus.cc:595] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.277621 11168 raft_consensus.cc:979] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a: : Instructing follower c778431b13784c7c81f3924b7f73e664 to start an election
I20240503 01:59:41.277724 11168 raft_consensus.cc:1067] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Signalling peer c778431b13784c7c81f3924b7f73e664 to start an election
W20240503 01:59:41.278276 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.278333 9254 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa"
dest_uuid: "c778431b13784c7c81f3924b7f73e664"
from {username='slave'} at 127.0.99.130:49973
I20240503 01:59:41.278499 9254 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:41.278609 9254 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.279841 9254 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:41.281113 9254 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [CANDIDATE]: Term 2 election: Requested vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:41.281366 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "c778431b13784c7c81f3924b7f73e664" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a"
I20240503 01:59:41.281425 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "c778431b13784c7c81f3924b7f73e664" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58"
I20240503 01:59:41.281489 7371 raft_consensus.cc:3032] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:41.281526 7371 raft_consensus.cc:729] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Running, Role: LEADER
I20240503 01:59:41.281536 7633 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.281673 7371 consensus_queue.cc:260] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:41.281847 7371 raft_consensus.cc:3037] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:41.282488 9190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:41.282570 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c778431b13784c7c81f3924b7f73e664 in term 2.
I20240503 01:59:41.282851 7371 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c778431b13784c7c81f3924b7f73e664 in term 2.
I20240503 01:59:41.283203 9190 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664, efeb87c28ee04b3da5c686665b0c5c58; no voters:
I20240503 01:59:41.283470 11096 raft_consensus.cc:2781] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:41.283682 11096 raft_consensus.cc:686] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [term 2 LEADER]: Becoming Leader. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Running, Role: LEADER
W20240503 01:59:41.283928 9190 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:41.283950 11096 consensus_queue.cc:237] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:41.286794 7090 catalog_manager.cc:5484] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 reported cstate change: term changed from 1 to 2, leader changed from bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) to c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "c778431b13784c7c81f3924b7f73e664" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.300531 12019 ts_tablet_manager.cc:902] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:41.301699 12019 tablet_copy_client.cc:308] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:41.302145 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 527af716906b42729952db28e790bafd from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:36477)
I20240503 01:59:41.302249 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 527af716906b42729952db28e790bafd from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:36477: session id = c778431b13784c7c81f3924b7f73e664-527af716906b42729952db28e790bafd
I20240503 01:59:41.303239 7523 tablet_copy_source_session.cc:205] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.304067 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.306344 12019 tablet_copy_client.cc:783] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.306589 12019 tablet_copy_client.cc:647] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.307694 12019 tablet_copy_client.cc:523] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.308861 12019 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:41.326376 12060 ts_tablet_manager.cc:902] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 01:59:41.327067 12060 tablet_copy_client.cc:308] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:41.327528 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 71f12a25a1db4650a6c37b5000a668c7 from peer c778431b13784c7c81f3924b7f73e664 ({username='slave'} at 127.0.99.135:48613)
I20240503 01:59:41.327666 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 71f12a25a1db4650a6c37b5000a668c7 from peer c778431b13784c7c81f3924b7f73e664 at {username='slave'} at 127.0.99.135:48613: session id = c778431b13784c7c81f3924b7f73e664-71f12a25a1db4650a6c37b5000a668c7
I20240503 01:59:41.328503 11747 raft_consensus.cc:1050] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb to VOTER
I20240503 01:59:41.328624 7392 tablet_copy_source_session.cc:205] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.328863 11747 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 01:59:41.329380 12060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:41.330806 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.330966 8991 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:41.331341 11105 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:41.331481 12060 tablet_copy_client.cc:783] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.331631 12060 tablet_copy_client.cc:647] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.332147 7240 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:41.332643 11105 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:41.333920 12060 tablet_copy_client.cc:523] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.335155 12060 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Bootstrap starting.
I20240503 01:59:41.335127 7502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf"
dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39322
I20240503 01:59:41.335299 7502 raft_consensus.cc:595] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.337404 11105 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.339530 8991 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.341362 7240 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 01:59:41.344040 7090 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.353248 12019 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.353601 12019 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:41.353827 12019 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.045s user 0.030s sys 0.000s
I20240503 01:59:41.354367 12019 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.354602 12019 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:41.354765 12019 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.355165 12019 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.356489 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-527af716906b42729952db28e790bafd received from {username='slave'} at 127.0.99.135:36477
I20240503 01:59:41.356560 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session c778431b13784c7c81f3924b7f73e664-527af716906b42729952db28e790bafd on tablet 527af716906b42729952db28e790bafd with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:41.368690 12060 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.368983 12060 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Bootstrap complete.
I20240503 01:59:41.369424 12060 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.001s
I20240503 01:59:41.369922 12060 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.370110 12060 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Initialized, Role: LEARNER
I20240503 01:59:41.370275 12060 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.370520 12060 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P c778431b13784c7c81f3924b7f73e664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.370962 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session c778431b13784c7c81f3924b7f73e664-71f12a25a1db4650a6c37b5000a668c7 received from {username='slave'} at 127.0.99.135:48613
I20240503 01:59:41.371048 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session c778431b13784c7c81f3924b7f73e664-71f12a25a1db4650a6c37b5000a668c7 on tablet 71f12a25a1db4650a6c37b5000a668c7 with peer c778431b13784c7c81f3924b7f73e664
I20240503 01:59:41.372648 8991 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:41.373283 11125 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:41.376073 7240 raft_consensus.cc:1203] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 01:59:41.389834 7632 raft_consensus.cc:1203] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 01:59:41.396191 7240 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 01:59:41.396667 7239 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:41.397661 7502 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:41.397701 9254 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:41.397953 11125 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:41.398025 11623 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:41.398507 11623 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:41.399374 7174 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66317a254eb644359f65610368d84d76. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.399886 11125 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.400633 11623 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:41.401268 9254 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:41.401340 7502 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:41.404083 7090 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 01:59:41.404062 8991 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb [term 2 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
W20240503 01:59:41.404729 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.406713 7090 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added, no longer has a pending config: VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129), VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132), VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134), VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133), NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.408210 7240 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.409694 7085 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 81086ff737c84d9d9f04743d31c6151b with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:41.412847 7502 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.414449 7371 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.414876 11057 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:41.414994 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.415760 7240 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 01:59:41.415904 7438 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 245cba4e5728491896e7995ca0d59b36. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.416087 11057 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.417022 11057 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.417865 7632 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.419055 7371 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.419914 7099 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.419910 7239 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.420325 7240 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 01:59:41.420423 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.420738 7502 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:41.421135 11557 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:41.422089 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4628afbfeff4f8bbaba5da1e89e4850. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.422251 11557 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:41.422849 12134 raft_consensus.cc:1013] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: attempt to promote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 01:59:41.423867 12134 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.425328 7240 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.426560 7502 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.427878 7092 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 9 to 10, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.428926 11557 raft_consensus.cc:1050] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 01:59:41.429273 11557 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:41.430212 7502 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:41.431697 7502 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:41.432857 7239 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 01:59:41.432894 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.433110 10287 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 01:59:41.433228 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): Couldn't send request to peer bed387c0ff9241688cdf45f364dc6f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40260acad62242c3a72197d2dafae187. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.433296 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.433598 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4628afbfeff4f8bbaba5da1e89e4850. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.433677 7240 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:41.433725 10287 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 01:59:41.433964 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.434106 11105 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:41.435757 12151 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:41.436414 12134 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.436681 7502 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.437722 7239 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:41.438572 7240 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.439728 7099 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:41.439965 7099 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 01:59:41.479954 11057 raft_consensus.cc:979] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb: : Instructing follower c778431b13784c7c81f3924b7f73e664 to start an election
I20240503 01:59:41.480103 11057 raft_consensus.cc:1067] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Signalling peer c778431b13784c7c81f3924b7f73e664 to start an election
I20240503 01:59:41.480664 9254 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf"
dest_uuid: "c778431b13784c7c81f3924b7f73e664"
from {username='slave'} at 127.0.99.131:59381
I20240503 01:59:41.480808 9254 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:41.480877 9254 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.481048 7502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf"
dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39420
I20240503 01:59:41.481171 7502 raft_consensus.cc:595] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:41.481961 9254 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:41.482583 9254 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [CANDIDATE]: Term 2 election: Requested vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145)
I20240503 01:59:41.483129 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "c778431b13784c7c81f3924b7f73e664" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb"
I20240503 01:59:41.483232 7502 raft_consensus.cc:3032] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:41.483269 7502 raft_consensus.cc:729] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Running, Role: LEADER
I20240503 01:59:41.483428 7502 consensus_queue.cc:260] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:41.483671 7502 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.484753 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c778431b13784c7c81f3924b7f73e664 in term 2.
I20240503 01:59:41.484728 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "c778431b13784c7c81f3924b7f73e664" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2"
I20240503 01:59:41.484843 7240 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.485837 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c778431b13784c7c81f3924b7f73e664 in term 2.
I20240503 01:59:41.486267 9191 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb, c778431b13784c7c81f3924b7f73e664; no voters:
I20240503 01:59:41.486430 11096 raft_consensus.cc:2781] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:41.486519 11096 raft_consensus.cc:686] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [term 2 LEADER]: Becoming Leader. State: Replica: c778431b13784c7c81f3924b7f73e664, State: Running, Role: LEADER
I20240503 01:59:41.486676 11096 consensus_queue.cc:237] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:41.487690 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "c778431b13784c7c81f3924b7f73e664" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e648eca559ed4e5ab24a1b98755aa3eb"
I20240503 01:59:41.487792 8991 raft_consensus.cc:3037] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:41.488607 8991 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P e648eca559ed4e5ab24a1b98755aa3eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c778431b13784c7c81f3924b7f73e664 in term 2.
I20240503 01:59:41.490727 7090 catalog_manager.cc:5484] T fb00fcf563b74243b7c955ddacf1d6cf P c778431b13784c7c81f3924b7f73e664 reported cstate change: term changed from 1 to 2, leader changed from 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) to c778431b13784c7c81f3924b7f73e664 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "c778431b13784c7c81f3924b7f73e664" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 01:59:41.499783 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.504753 7502 raft_consensus.cc:1203] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
W20240503 01:59:41.544678 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.570103 7240 raft_consensus.cc:1203] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20240503 01:59:41.592173 11890 ts_tablet_manager.cc:902] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
W20240503 01:59:41.592582 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.592798 11890 tablet_copy_client.cc:308] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 01:59:41.593302 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:50331)
I20240503 01:59:41.593415 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 5f35ccb9c21b4c10b9187596d2f6929c from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:50331: session id = 4356d1cb1050473a8d9e50ff09a24dc2-5f35ccb9c21b4c10b9187596d2f6929c
I20240503 01:59:41.594347 7392 tablet_copy_source_session.cc:205] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.595011 11890 ts_tablet_manager.cc:1875] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 01:59:41.596922 11890 ts_tablet_manager.cc:1888] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 01:59:41.597081 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f35ccb9c21b4c10b9187596d2f6929c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.598064 11890 tablet_copy_client.cc:783] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.598187 11890 tablet_copy_client.cc:647] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.599272 11890 tablet_copy_client.cc:523] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.600746 11890 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 01:59:41.624737 11890 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.625038 11890 tablet_bootstrap.cc:492] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 01:59:41.625185 11890 ts_tablet_manager.cc:1367] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20240503 01:59:41.625535 11890 raft_consensus.cc:348] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:41.625659 11890 raft_consensus.cc:729] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 01:59:41.625753 11890 consensus_queue.cc:260] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 01:59:41.626011 11890 ts_tablet_manager.cc:1398] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:41.626668 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-5f35ccb9c21b4c10b9187596d2f6929c received from {username='slave'} at 127.0.99.129:50331
I20240503 01:59:41.626735 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-5f35ccb9c21b4c10b9187596d2f6929c on tablet 5f35ccb9c21b4c10b9187596d2f6929c with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 01:59:41.647363 7371 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:41.647790 12133 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:41.680377 12060 ts_tablet_manager.cc:902] T 81086ff737c84d9d9f04743d31c6151b P c778431b13784c7c81f3924b7f73e664: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
F20240503 01:59:41.680559 12060 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.queue_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.queue_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_cpu_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_cpu_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_wall_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_wall_time_us) (tablet-download-pool-66317a254eb644359f65610368d84d76.queue_time_us, tablet-download-pool-66317a254eb644359f65610368d84d76.queue_time_us) (tablet-download-pool-66317a254eb644359f65610368d84d76.run_cpu_time_us, tablet-download-pool-66317a254eb644359f65610368d84d76.run_cpu_time_us) (tablet-download-pool-66317a254eb644359f65610368d84d76.run_wall_time_us, tablet-download-pool-66317a254eb644359f65610368d84d76.run_wall_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.queue_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.queue_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_cpu_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_cpu_time_us) (tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_wall_time_us, tablet-download-pool-9796308eacd94fb5ad3cb524aa590cfa.run_wall_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.queue_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_cpu_time_us) (tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us, tablet-download-pool-a0b1a4b2ede6493f9e4e094649b4b57d.run_wall_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.queue_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.queue_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_cpu_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_cpu_time_us) (tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_wall_time_us, tablet-download-pool-a432cd5fe20941d6bc73dae386d85170.run_wall_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.queue_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.queue_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_cpu_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_cpu_time_us) (tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_wall_time_us, tablet-download-pool-be1f4d2622c6452b869f88a301a5e9bc.run_wall_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.queue_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.queue_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_cpu_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_cpu_time_us) (tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_wall_time_us, tablet-download-pool-c95d3c29bfbd4899b27b770bf9d3f4a6.run_wall_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.queue_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.queue_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_cpu_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_cpu_time_us) (tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_wall_time_us, tablet-download-pool-cc68f31e6ef8411e96d46b2419bf9676.run_wall_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.queue_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.queue_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_cpu_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_cpu_time_us) (tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_wall_time_us, tablet-download-pool-f0539dded7554723b9e918e8cdfbab20.run_wall_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.queue_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.queue_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_cpu_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_cpu_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_wall_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7eff9a6e4dcd google::LogMessage::Fail() at ??:0
I20240503 01:59:41.684260 7371 raft_consensus.cc:1261] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Refusing update from remote peer c778431b13784c7c81f3924b7f73e664: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:41.684764 11096 consensus_queue.cc:1035] T 9796308eacd94fb5ad3cb524aa590cfa P c778431b13784c7c81f3924b7f73e664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7eff9a6e8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7eff9a6e47cc google::LogMessage::Flush() at ??:0
@ 0x7eff9a6e5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7eff9bb4e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7eff9bb32f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7eff9bb30efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7eff9e6e377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7eff9e6eca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7eff9e76d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7eff9e7814f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 01:59:41.707195 9254 raft_consensus.cc:1203] T 527af716906b42729952db28e790bafd P c778431b13784c7c81f3924b7f73e664 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
@ 0x7eff9e76c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7eff9e77bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55a5b0c6775c std::function<>::operator()() at ??:0
@ 0x7eff9bb35e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7eff9bb366ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7eff9bb3803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55a5b0c6775c std::function<>::operator()() at ??:0
@ 0x7eff9bb25807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7eff9c3636db start_thread at ??:0
@ 0x7eff9989b71f clone at ??:0
W20240503 01:59:41.810300 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:41.846063 9522 raft_consensus.cc:1050] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 01:59:41.846556 9522 consensus_queue.cc:237] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 01:59:41.847815 7632 raft_consensus.cc:1261] T a432cd5fe20941d6bc73dae386d85170 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 01:59:41.848217 9522 consensus_queue.cc:1035] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 01:59:41.848413 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.850529 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:41.909680 7438 connection.cc:535] client connection to 127.0.99.135:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.910059 7307 connection.cc:535] client connection to 127.0.99.135:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.910712 7569 connection.cc:535] client connection to 127.0.99.135:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:41.911168 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.911779 11892 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:41.910517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43981 (local address 127.0.99.131:35163)
0503 01:59:41.911333 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:41.911375 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":744}
W20240503 01:59:41.912119 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection reset by peer (error 104)
W20240503 01:59:41.912345 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:41.912413 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.912459 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.912503 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:41.923141 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:41.933501 10287 raft_consensus.cc:1050] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 01:59:41.933933 10287 consensus_queue.cc:237] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
W20240503 01:59:41.936646 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.936844 7240 raft_consensus.cc:1261] T c95d3c29bfbd4899b27b770bf9d3f4a6 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 01:59:41.937268 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.937482 12134 consensus_queue.cc:1035] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:41.943799 12438 ts_tablet_manager.cc:902] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 01:59:41.944677 12438 tablet_copy_client.cc:308] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 01:59:41.946061 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 40260acad62242c3a72197d2dafae187 from peer bed387c0ff9241688cdf45f364dc6f5a ({username='slave'} at 127.0.99.130:46341)
I20240503 01:59:41.946178 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 40260acad62242c3a72197d2dafae187 from peer bed387c0ff9241688cdf45f364dc6f5a at {username='slave'} at 127.0.99.130:46341: session id = bed387c0ff9241688cdf45f364dc6f5a-40260acad62242c3a72197d2dafae187
I20240503 01:59:41.947093 7523 tablet_copy_source_session.cc:205] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:41.948596 12438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.951174 12438 tablet_copy_client.cc:783] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:41.951364 12438 tablet_copy_client.cc:647] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:41.952900 12438 tablet_copy_client.cc:523] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.954715 12438 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
W20240503 01:59:41.962484 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.977790 12467 ts_tablet_manager.cc:902] T 66317a254eb644359f65610368d84d76 P e648eca559ed4e5ab24a1b98755aa3eb: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
F20240503 01:59:41.978111 12467 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.queue_time_us, tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.queue_time_us) (tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.run_cpu_time_us, tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.run_cpu_time_us) (tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.run_wall_time_us, tablet-download-pool-0d00b8fcb29f47298643903ec0cba606.run_wall_time_us) (tablet-download-pool-18468345c7924d4f88366524b56e8ab3.queue_time_us, tablet-download-pool-18468345c7924d4f88366524b56e8ab3.queue_time_us) (tablet-download-pool-18468345c7924d4f88366524b56e8ab3.run_cpu_time_us, tablet-download-pool-18468345c7924d4f88366524b56e8ab3.run_cpu_time_us) (tablet-download-pool-18468345c7924d4f88366524b56e8ab3.run_wall_time_us, tablet-download-pool-18468345c7924d4f88366524b56e8ab3.run_wall_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.queue_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.queue_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_cpu_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_cpu_time_us) (tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_wall_time_us, tablet-download-pool-27c539b143674bdd8e3e5aaa4b78c1e9.run_wall_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.queue_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.queue_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_cpu_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_cpu_time_us) (tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_wall_time_us, tablet-download-pool-3b15d48b2fa04269b98c6f801f54df4f.run_wall_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.queue_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_cpu_time_us) (tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us, tablet-download-pool-527af716906b42729952db28e790bafd.run_wall_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.queue_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.queue_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_cpu_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_cpu_time_us) (tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_wall_time_us, tablet-download-pool-5f35ccb9c21b4c10b9187596d2f6929c.run_wall_time_us) (tablet-download-pool-66317a254eb644359f65610368d84d76.queue_time_us, tablet-download-pool-66317a254eb644359f65610368d84d76.queue_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.queue_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_cpu_time_us) (tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us, tablet-download-pool-6649e1feab5b40738003119a9122f07a.run_wall_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.queue_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_cpu_time_us) (tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us, tablet-download-pool-71f12a25a1db4650a6c37b5000a668c7.run_wall_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.queue_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_cpu_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_cpu_time_us) (tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_wall_time_us, tablet-download-pool-81086ff737c84d9d9f04743d31c6151b.run_wall_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.queue_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_cpu_time_us) (tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us, tablet-download-pool-8ad24b47690b4263bdf4a9157db7c17f.run_wall_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.queue_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.queue_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_cpu_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_cpu_time_us) (tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_wall_time_us, tablet-download-pool-8ebe541161c84bd6b11bcd3d13d3bd2e.run_wall_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.queue_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_cpu_time_us) (tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us, tablet-download-pool-97ffb59d3ab64d03aca5d30a7c3be1e1.run_wall_time_us) (tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.queue_time_us, tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.queue_time_us) (tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.run_cpu_time_us, tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.run_cpu_time_us) (tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.run_wall_time_us, tablet-download-pool-c18129344d124a5f8f688ec35ef69a55.run_wall_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.queue_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.queue_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_cpu_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_cpu_time_us) (tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_wall_time_us, tablet-download-pool-fb00fcf563b74243b7c955ddacf1d6cf.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 01:59:41.982277 12438 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7fb123f60dcd google::LogMessage::Fail() at ??:0
I20240503 01:59:41.982709 12438 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap complete.
I20240503 01:59:41.982916 12438 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.006s
I20240503 01:59:41.983397 12438 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:41.983534 12438 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: LEARNER
I20240503 01:59:41.983670 12438 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:41.984514 12438 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:41.985049 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session bed387c0ff9241688cdf45f364dc6f5a-40260acad62242c3a72197d2dafae187 received from {username='slave'} at 127.0.99.130:46341
I20240503 01:59:41.985126 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session bed387c0ff9241688cdf45f364dc6f5a-40260acad62242c3a72197d2dafae187 on tablet 40260acad62242c3a72197d2dafae187 with peer bed387c0ff9241688cdf45f364dc6f5a
@ 0x7fb123f64b93 google::LogMessage::SendToLog() at ??:0
I20240503 01:59:41.989276 9522 raft_consensus.cc:1050] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 01:59:41.989665 9522 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 01:59:41.991182 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 01:59:41.991623 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 01:59:41.992003 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 01:59:41.993897 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 01:59:41.994035 7502 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20240503 01:59:41.994455 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:41.994485 11086 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 01:59:41.996126 9522 raft_consensus.cc:1013] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: attempt to promote peer 4356d1cb1050473a8d9e50ff09a24dc2: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
@ 0x7fb123f607cc google::LogMessage::Flush() at ??:0
@ 0x7fb123f61f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb1253ca044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fb1253aef98 kudu::ThreadPool::ThreadPool() at ??:0
W20240503 01:59:42.015969 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
@ 0x7fb1253acefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fb127f5f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 01:59:42.029111 11168 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
@ 0x7fb127f68a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 01:59:42.032191 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.033214 7502 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
@ 0x7fb127fe976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 01:59:42.034490 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.037343 7094 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 14 to 15, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.037997 10236 raft_consensus.cc:1050] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 01:59:42.038501 10236 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
@ 0x7fb127ffd4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 01:59:42.040920 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20240503 01:59:42.041075 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.041445 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 01:59:42.042565 7502 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
@ 0x7fb127fe8226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 01:59:42.043005 9522 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 01:59:42.043190 8991 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 01:59:42.043682 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 01:59:42.045454 11086 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.046054 8991 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
@ 0x7fb127ff7c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d26993775c std::function<>::operator()() at ??:0
I20240503 01:59:42.047922 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.047997 7502 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
@ 0x7fb1253b1e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1253b26ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb1253b403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
W20240503 01:59:42.058908 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x55d26993775c std::function<>::operator()() at ??:0
@ 0x7fb1253a1807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb125bdf6db start_thread at ??:0
@ 0x7fb12311771f clone at ??:0
I20240503 01:59:42.077492 7098 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P e648eca559ed4e5ab24a1b98755aa3eb reported cstate change: config changed from index 15 to 16, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.079447 7221 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.083850 7350 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:42.087039 7613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 01:59:42.087604 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:42.092491 7483 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:42.094548 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 01:59:42.096956 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240503 01:59:42.097352 11169 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240503 01:59:42.097754 7306 consensus_peers.cc:485] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.098740 7502 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240503 01:59:42.099154 11169 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 01:59:42.101321 11169 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.103201 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.103246 7502 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.104822 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6649e1feab5b40738003119a9122f07a with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:42.106462 7091 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 16 to 17, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.111932 7371 consensus_queue.cc:237] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 01:59:42.113909 7240 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20240503 01:59:42.114332 7502 raft_consensus.cc:1261] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20240503 01:59:42.114727 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 01:59:42.115051 11168 consensus_queue.cc:1035] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 01:59:42.116605 10236 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.116945 7240 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.122785 7094 catalog_manager.cc:5484] T 6649e1feab5b40738003119a9122f07a P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 17 to 18, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 01:59:42.123420 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6649e1feab5b40738003119a9122f07a with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:42.123909 7502 raft_consensus.cc:2932] T 6649e1feab5b40738003119a9122f07a P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
W20240503 01:59:42.139932 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.147588 7240 raft_consensus.cc:1203] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240503 01:59:42.149617 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.162591 7221 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.167527 7350 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:42.185899 7483 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:42.191314 7613 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 01:59:42.201702 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.227160 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:42.229655 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:42.307855 7305 connection.cc:535] client connection to 127.0.99.133:34145 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:42.309039 7436 connection.cc:535] client connection to 127.0.99.133:34145 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:42.309058 7084 connection.cc:535] server connection from 127.0.99.133:43325 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:42.309362 7083 connection.cc:535] client connection to 127.0.99.133:34145 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 01:59:42.309685 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:42.310366 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.310451 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.310920 11732 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:42.308605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:34145 (local address 127.0.99.130:40433)
0503 01:59:42.308752 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:42.308766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:42.308881 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:42.310312 (+ 1431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
W20240503 01:59:42.311069 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.311121 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.311156 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.323581 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:42.328455 9436 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8Tfalf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 01:59:35.361799 9441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.362087 9441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.366189 9441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:36.150379 9441 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:36.600068 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.600317 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.600394 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.600437 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.600498 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.600528 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.600554 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.600584 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.600625 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.600651 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.600690 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.600716 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.600746 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.600776 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.600802 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:36.717264 9441 rebalancer_tool.cc:1074] tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:36.804714 9441 rebalancer_tool.cc:1074] tablet 27c539b143674bdd8e3e5aaa4b78c1e9: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:36.862759 9441 rebalancer_tool.cc:1074] tablet 71f12a25a1db4650a6c37b5000a668c7: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:36.939764 9441 rebalancer_tool.cc:1074] tablet 8ad24b47690b4263bdf4a9157db7c17f: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.055272 9441 rebalancer_tool.cc:1074] tablet cc68f31e6ef8411e96d46b2419bf9676: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.223385 9441 rebalancer_tool.cc:1074] tablet f0539dded7554723b9e918e8cdfbab20: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.274287 9441 rebalancer_tool.cc:1074] tablet ec179e5e683d4ba099c459331ea5e578: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.303323 9441 rebalancer_tool.cc:1074] tablet 6649e1feab5b40738003119a9122f07a: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.329914 9441 rebalancer_tool.cc:1074] tablet a0b1a4b2ede6493f9e4e094649b4b57d: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.389015 9441 rebalancer_tool.cc:1074] tablet c95d3c29bfbd4899b27b770bf9d3f4a6: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.403718 9441 rebalancer_tool.cc:1084] tablet 0d00b8fcb29f47298643903ec0cba606: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move ignored: Illegal state: tablet 0d00b8fcb29f47298643903ec0cba606: replica e648eca559ed4e5ab24a1b98755aa3eb is already present
I20240503 01:59:37.403793 9441 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:38.454954 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:38.455077 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:38.455130 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.455180 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:38.455225 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:38.455277 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.455324 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:38.455369 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:38.455456 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:38.455516 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:38.567246 9441 rebalancer_tool.cc:1074] tablet a432cd5fe20941d6bc73dae386d85170: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:38.744133 9441 rebalancer_tool.cc:1074] tablet c18129344d124a5f8f688ec35ef69a55: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:39.036862 9441 rebalancer_tool.cc:1084] tablet 9796308eacd94fb5ad3cb524aa590cfa: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move ignored: Illegal state: tablet 9796308eacd94fb5ad3cb524aa590cfa: replica 72577fce0313400ea862bf873738af54 is already present
I20240503 01:59:39.036959 9441 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:40.477479 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:40.477667 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server c778431b13784c7c81f3924b7f73e664 to server 5b46f0d4b665497b8d7e57bf4d5f36cb
W20240503 01:59:40.477728 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server bed387c0ff9241688cdf45f364dc6f5a
W20240503 01:59:40.477813 9441 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:40.531699 9441 rebalancer_tool.cc:1074] tablet 0d00b8fcb29f47298643903ec0cba606: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '' move scheduled
I20240503 01:59:40.614029 9441 rebalancer_tool.cc:1074] tablet 9796308eacd94fb5ad3cb524aa590cfa: 'bed387c0ff9241688cdf45f364dc6f5a' -> '' move scheduled
I20240503 01:59:40.760293 9441 rebalancer_tool.cc:1074] tablet b4628afbfeff4f8bbaba5da1e89e4850: '4356d1cb1050473a8d9e50ff09a24dc2' -> '5b46f0d4b665497b8d7e57bf4d5f36cb' move scheduled
I20240503 01:59:40.871364 9441 rebalancer_tool.cc:1074] tablet 527af716906b42729952db28e790bafd: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:41.111271 9441 rebalancer_tool.cc:1074] tablet 5f35ccb9c21b4c10b9187596d2f6929c: 'bed387c0ff9241688cdf45f364dc6f5a' -> '4356d1cb1050473a8d9e50ff09a24dc2' move scheduled
I20240503 01:59:41.202971 9441 rebalancer_tool.cc:1136] tablet 71f12a25a1db4650a6c37b5000a668c7: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move completed: Incomplete: tablet 71f12a25a1db4650a6c37b5000a668c7, TS 4356d1cb1050473a8d9e50ff09a24dc2 -> TS 72577fce0313400ea862bf873738af54 move failed, target replica disappeared
I20240503 01:59:41.239887 9441 rebalancer_tool.cc:1136] tablet cc68f31e6ef8411e96d46b2419bf9676: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move completed: OK
I20240503 01:59:41.255236 9441 rebalancer_tool.cc:1136] tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move completed: OK
I20240503 01:59:41.263434 9441 rebalancer_tool.cc:1136] tablet 27c539b143674bdd8e3e5aaa4b78c1e9: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move completed: OK
I20240503 01:59:41.271530 9441 rebalancer_tool.cc:1136] tablet ec179e5e683d4ba099c459331ea5e578: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move completed: OK
I20240503 01:59:41.282212 9441 rebalancer_tool.cc:1136] tablet 6649e1feab5b40738003119a9122f07a: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.298342 9441 rebalancer_tool.cc:1136] tablet a0b1a4b2ede6493f9e4e094649b4b57d: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move completed: Incomplete: tablet a0b1a4b2ede6493f9e4e094649b4b57d, TS 5b46f0d4b665497b8d7e57bf4d5f36cb -> TS c778431b13784c7c81f3924b7f73e664 move failed, target replica disappeared
I20240503 01:59:41.306573 9441 rebalancer_tool.cc:1136] tablet c95d3c29bfbd4899b27b770bf9d3f4a6: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move completed: OK
I20240503 01:59:41.320900 9441 rebalancer_tool.cc:1136] tablet 8ad24b47690b4263bdf4a9157db7c17f: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.329540 9441 rebalancer_tool.cc:1136] tablet a432cd5fe20941d6bc73dae386d85170: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move completed: Incomplete: tablet a432cd5fe20941d6bc73dae386d85170, TS 4356d1cb1050473a8d9e50ff09a24dc2 -> TS 72577fce0313400ea862bf873738af54 move failed, target replica disappeared
I20240503 01:59:41.352377 9441 rebalancer_tool.cc:1136] tablet 0d00b8fcb29f47298643903ec0cba606: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '' move completed: OK
I20240503 01:59:41.381981 9441 rebalancer_tool.cc:1136] tablet c18129344d124a5f8f688ec35ef69a55: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.402824 9441 rebalancer_tool.cc:1074] tablet 66317a254eb644359f65610368d84d76: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:41.418478 9441 rebalancer_tool.cc:1074] tablet 245cba4e5728491896e7995ca0d59b36: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:41.438149 9441 rebalancer_tool.cc:1074] tablet 40260acad62242c3a72197d2dafae187: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
W20240503 01:59:41.911597 12371 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:41.758602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43981 (local address 127.0.0.1:51548)
0503 01:59:41.758809 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:41.758821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:41.758909 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:41.910613 (+151704us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20240503 01:59:41.912416 9441 rebalancer_tool.cc:1123] tablet 9796308eacd94fb5ad3cb524aa590cfa: 'bed387c0ff9241688cdf45f364dc6f5a' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 01:59:41.919294 9441 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:42.308040 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:42.070880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:34145 (local address 127.0.0.1:52948)
0503 01:59:42.077918 (+ 7038us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:42.077926 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:42.077979 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:42.307853 (+229874us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7005}
W20240503 01:59:42.310062 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:42.308612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:34145 (local address 127.0.0.1:52968)
0503 01:59:42.308763 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:42.308779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:42.308876 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:42.309985 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":57}
Illegal state: tablet server c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): unacceptable health status UNAVAILABLE
I20240503 01:59:42.329571 9439 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task8Tfalf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 01:59:35.386847 9442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.387110 9442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.395797 9442 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:36.523401 9442 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:36.844292 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.844489 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.844552 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.844580 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.844640 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 824ac75858614f5a9292bf5cfabf8c80: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.844684 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.844719 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.844749 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:36.844790 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.844832 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.844888 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.844938 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 37323b4f15ad49a1b2a00e2eb771c02f: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:36.844982 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 525842a82f914047b4e7a020aa11ca26: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.845103 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:36.845173 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:36.845222 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:36.915345 9442 rebalancer_tool.cc:1074] tablet 81086ff737c84d9d9f04743d31c6151b: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.019413 9442 rebalancer_tool.cc:1074] tablet 66317a254eb644359f65610368d84d76: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.081827 9442 rebalancer_tool.cc:1074] tablet 3b15d48b2fa04269b98c6f801f54df4f: 'efeb87c28ee04b3da5c686665b0c5c58' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.107509 9442 rebalancer_tool.cc:1074] tablet 71f12a25a1db4650a6c37b5000a668c7: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.244482 9442 rebalancer_tool.cc:1074] tablet cc68f31e6ef8411e96d46b2419bf9676: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:37.260896 9442 rebalancer_tool.cc:1074] tablet f0539dded7554723b9e918e8cdfbab20: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:37.331804 9442 rebalancer_tool.cc:1074] tablet 5f35ccb9c21b4c10b9187596d2f6929c: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:37.369153 9442 rebalancer_tool.cc:1074] tablet 527af716906b42729952db28e790bafd: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:37.436729 9442 rebalancer_tool.cc:1084] tablet be1f4d2622c6452b869f88a301a5e9bc: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '72577fce0313400ea862bf873738af54' move ignored: Illegal state: tablet be1f4d2622c6452b869f88a301a5e9bc: replica 72577fce0313400ea862bf873738af54 is already present
I20240503 01:59:37.436812 9442 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:38.404640 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8d5ead3760884fe38d78dd9dcf4fc139: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server 5b46f0d4b665497b8d7e57bf4d5f36cb
W20240503 01:59:38.404712 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.404857 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:38.404946 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server bed387c0ff9241688cdf45f364dc6f5a to server c778431b13784c7c81f3924b7f73e664
I20240503 01:59:38.523033 9442 rebalancer_tool.cc:1074] tablet 9796308eacd94fb5ad3cb524aa590cfa: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move scheduled
I20240503 01:59:38.608086 9442 rebalancer_tool.cc:1074] tablet fb00fcf563b74243b7c955ddacf1d6cf: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
I20240503 01:59:38.808516 9442 rebalancer_tool.cc:1074] tablet ec179e5e683d4ba099c459331ea5e578: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:39.070883 9442 rebalancer_tool.cc:1074] tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move scheduled
I20240503 01:59:39.311087 9442 rebalancer_tool.cc:1084] tablet c95d3c29bfbd4899b27b770bf9d3f4a6: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 01:59:39.311170 9442 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:40.353854 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server c8ecb5fc8d804c9b9ca42ee4b2377f4e
W20240503 01:59:40.353947 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3ea4dc3283bd46ffb0691a36b400dbb1: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server 5b46f0d4b665497b8d7e57bf4d5f36cb
W20240503 01:59:40.354044 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 325e5f6cef5d4f3687726b6b27765419: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:40.354079 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 135581cd6a7f43aeb219da3a660f73dc: could not find any suitable replica to move from server efeb87c28ee04b3da5c686665b0c5c58 to server c778431b13784c7c81f3924b7f73e664
W20240503 01:59:40.354130 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 304a6c7b52ea465fb77b2bd88c2fda58: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
W20240503 01:59:40.354246 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1fa7c6f256e94bc9ad446cf003876203: could not find any suitable replica to move from server 4356d1cb1050473a8d9e50ff09a24dc2 to server 72577fce0313400ea862bf873738af54
W20240503 01:59:40.354295 9442 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1347ea81a2854d61b22e8e363d091380: could not find any suitable replica to move from server 5b46f0d4b665497b8d7e57bf4d5f36cb to server e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:40.385294 9442 rebalancer_tool.cc:1074] tablet 68791b3adccd4168b55a2da4423232d7: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '' move scheduled
I20240503 01:59:40.481191 9442 rebalancer_tool.cc:1074] tablet c95d3c29bfbd4899b27b770bf9d3f4a6: 'c778431b13784c7c81f3924b7f73e664' -> '4356d1cb1050473a8d9e50ff09a24dc2' move scheduled
I20240503 01:59:40.620533 9442 rebalancer_tool.cc:1074] tablet a432cd5fe20941d6bc73dae386d85170: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'efeb87c28ee04b3da5c686665b0c5c58' move scheduled
I20240503 01:59:40.760141 9442 rebalancer_tool.cc:1074] tablet 6649e1feab5b40738003119a9122f07a: 'e648eca559ed4e5ab24a1b98755aa3eb' -> '5b46f0d4b665497b8d7e57bf4d5f36cb' move scheduled
I20240503 01:59:40.882804 9442 rebalancer_tool.cc:1074] tablet be1f4d2622c6452b869f88a301a5e9bc: 'c778431b13784c7c81f3924b7f73e664' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move scheduled
I20240503 01:59:41.115172 9442 rebalancer_tool.cc:1074] tablet 40260acad62242c3a72197d2dafae187: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'bed387c0ff9241688cdf45f364dc6f5a' move scheduled
I20240503 01:59:41.254180 9442 rebalancer_tool.cc:1136] tablet 71f12a25a1db4650a6c37b5000a668c7: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.263499 9442 rebalancer_tool.cc:1136] tablet cc68f31e6ef8411e96d46b2419bf9676: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move completed: OK
I20240503 01:59:41.294147 9442 rebalancer_tool.cc:1136] tablet 5f35ccb9c21b4c10b9187596d2f6929c: 'efeb87c28ee04b3da5c686665b0c5c58' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.299860 9442 rebalancer_tool.cc:1136] tablet 81086ff737c84d9d9f04743d31c6151b: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c8ecb5fc8d804c9b9ca42ee4b2377f4e' move completed: OK
I20240503 01:59:41.308249 9442 rebalancer_tool.cc:1136] tablet f0539dded7554723b9e918e8cdfbab20: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c778431b13784c7c81f3924b7f73e664' move completed: OK
I20240503 01:59:41.314565 9442 rebalancer_tool.cc:1136] tablet 66317a254eb644359f65610368d84d76: 'bed387c0ff9241688cdf45f364dc6f5a' -> 'c778431b13784c7c81f3924b7f73e664' move completed: OK
I20240503 01:59:41.320324 9442 rebalancer_tool.cc:1136] tablet 527af716906b42729952db28e790bafd: '4356d1cb1050473a8d9e50ff09a24dc2' -> '72577fce0313400ea862bf873738af54' move completed: OK
I20240503 01:59:41.378614 9442 rebalancer_tool.cc:1136] tablet 3b15d48b2fa04269b98c6f801f54df4f: 'efeb87c28ee04b3da5c686665b0c5c58' -> '72577fce0313400ea862bf873738af54' move completed: OK
I20240503 01:59:41.384443 9442 rebalancer_tool.cc:1136] tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'e648eca559ed4e5ab24a1b98755aa3eb' move completed: OK
I20240503 01:59:41.390750 9442 rebalancer_tool.cc:1136] tablet 68791b3adccd4168b55a2da4423232d7: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> '' move completed: OK
I20240503 01:59:41.428063 9442 rebalancer_tool.cc:1074] tablet b4628afbfeff4f8bbaba5da1e89e4850: '4356d1cb1050473a8d9e50ff09a24dc2' -> 'c778431b13784c7c81f3924b7f73e664' move scheduled
W20240503 01:59:41.911603 12365 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:41.753030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43981 (local address 127.0.0.1:51544)
0503 01:59:41.753802 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:41.753814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:41.753885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:41.910817 (+156932us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":111,"threads_started":1}
I20240503 01:59:41.912510 9442 rebalancer_tool.cc:1123] tablet 9796308eacd94fb5ad3cb524aa590cfa: 'bed387c0ff9241688cdf45f364dc6f5a' -> '72577fce0313400ea862bf873738af54' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 01:59:41.922176 9442 rebalancer_tool.cc:1123] tablet fb00fcf563b74243b7c955ddacf1d6cf: '5b46f0d4b665497b8d7e57bf4d5f36cb' -> 'c778431b13784c7c81f3924b7f73e664' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:41.941366 9442 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:42.309258 12560 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 01:59:42.157580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:34145 (local address 127.0.0.1:52962)
0503 01:59:42.165612 (+ 8032us) negotiation.cc:107] Waiting for socket to connect
0503 01:59:42.165624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 01:59:42.165688 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 01:59:42.309136 (+143448us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7972}
Illegal state: tablet server 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): unacceptable health status UNAVAILABLE
W20240503 01:59:42.388731 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:42.408087 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:42.409273 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:42.424053 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.424386 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.430972 7371 raft_consensus.cc:1203] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 01:59:42.443290 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.448679 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.529495 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:42.580049 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.589614 9522 raft_consensus.cc:1050] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 01:59:42.590032 9522 consensus_queue.cc:237] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 01:59:42.591606 7240 raft_consensus.cc:1261] T 5f35ccb9c21b4c10b9187596d2f6929c P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:42.592145 9522 consensus_queue.cc:1035] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 01:59:42.592479 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.592662 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.697052 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:42.698100 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:42.784135 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:42.820677 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:42.825387 12588 raft_consensus.cc:1050] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER bed387c0ff9241688cdf45f364dc6f5a to VOTER
I20240503 01:59:42.825891 12588 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 01:59:42.827724 7371 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:42.827769 7240 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:42.828130 12614 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 01:59:42.828261 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:42.828253 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.828557 12614 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:42.830722 12650 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:42.831300 7240 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:42.832578 7371 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 01:59:42.833364 7094 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:42.909899 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:42.934890 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:42.942224 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:43.047031 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:43.051267 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:43.093461 12133 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.093588 12133 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:43.094081 12133 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:43.094412 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
W20240503 01:59:43.095085 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:43.095180 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 01:59:43.095271 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:43.095319 12133 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:43.155818 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:43.163004 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:43.186125 11169 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 01:59:43.186285 11169 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:43.186724 11169 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:43.187144 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:43.187306 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 01:59:43.187865 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:43.187981 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:43.188068 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:43.188205 10236 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:43.191452 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.242055 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.254081 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:43.289860 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:43.295414 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.353613 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.354200 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:43.364208 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.410161 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:43.410187 12650 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.410292 12650 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:43.410706 12650 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:43.411013 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:43.411180 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 01:59:43.411644 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:43.411721 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:43.411770 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:43.411921 12650 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:43.423812 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.436384 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:43.450949 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.477525 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.528519 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:43.599987 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.669840 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:43.669880 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.669936 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.694723 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:43.739763 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:43.779158 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:43.822906 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.827471 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:43.856869 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.879526 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.894145 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:43.981403 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:44.000551 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.005648 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.028297 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.115314 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:44.168454 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:44.197865 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.204936 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.217438 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:44.227803 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.243189 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.256932 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.332867 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.340811 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.344928 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:44.354373 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.424500 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.492633 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:44.505087 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.555344 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:44.609048 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.620430 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:44.629675 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.659144 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.675674 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.675829 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.689486 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:44.738328 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.818789 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.836383 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.871685 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:44.900653 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:44.938448 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:44.941483 12650 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.941613 12650 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:44.942111 12650 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:44.942454 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:44.942623 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 01:59:44.943141 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:44.943212 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:44.943259 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:44.943387 12650 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:44.946758 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.984532 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:44.985791 10236 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 01:59:44.985894 10236 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:44.986375 10236 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:44.986685 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:44.986861 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 01:59:44.987375 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:44.987469 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:44.987592 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:44.987718 10236 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:45.034786 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.070736 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:45.095436 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:45.103268 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.128552 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:45.183344 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:45.201890 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.287160 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.291965 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:45.325301 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.336588 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:45.390158 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.466177 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.482089 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:45.548925 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:45.561249 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:45.563578 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:45.580962 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.618816 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:45.621244 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:45.636168 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:45.700776 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.701736 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.762806 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.815112 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:45.815119 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.823937 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 187 similar messages]
W20240503 01:59:45.824596 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.829138 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:45.896626 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 01:59:45.897207 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:45.960150 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.034097 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:46.040310 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:46.093735 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:46.105243 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.113634 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240503 01:59:46.129900 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.138238 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 01:59:46.225268 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:46.251240 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:46.275060 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.307484 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.364281 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.378759 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.398782 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:46.406702 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240503 01:59:46.421648 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.425086 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:46.436380 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.512063 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.537628 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:46.596089 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:46.652086 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:46.696630 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:46.717262 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.766350 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:46.776015 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:46.789144 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.797010 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.804929 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 01:59:46.866160 12687 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.866307 12687 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:46.866873 12687 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:46.867274 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:46.867430 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 01:59:46.867964 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:46.868121 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:46.868194 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:46.868337 12687 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:46.889585 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.899485 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:46.900861 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:47.015269 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.042416 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.047266 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.054061 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.071745 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.080135 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.100244 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.104300 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.193976 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.194974 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:47.240818 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.240940 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.254861 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.266664 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:47.288833 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.368532 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:47.380875 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.454357 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.458446 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:47.459287 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.466372 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:47.478696 10236 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 01:59:47.478855 10236 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:47.479351 10236 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:47.479735 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:47.479900 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 01:59:47.480406 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.480496 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:47.480556 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:47.480669 10236 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:47.594379 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:47.626228 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.730854 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:47.811266 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.852545 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:47.864310 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.873123 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.897374 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.935362 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:47.967458 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:47.975991 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:47.988070 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:48.004202 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.042327 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.090488 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.162552 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:48.238231 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.256029 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.282631 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.292902 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:48.324431 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.400012 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:48.437753 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.441264 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.456893 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.459705 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.460619 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:48.546339 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.550155 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.571221 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.571307 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:48.597847 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:48.611794 12687 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.611963 12687 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:48.612462 12687 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:48.612952 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:48.613142 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 01:59:48.613667 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.613847 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 01:59:48.613907 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:48.614020 12687 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:48.615993 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.674088 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.697086 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:48.811887 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.812676 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.824930 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:48.899588 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:48.942500 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.009120 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:49.019483 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:49.024228 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.046849 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.114382 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.138850 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.143378 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:49.148114 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.250874 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.271991 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:49.279592 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.296262 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.404521 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.421665 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.461239 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.514707 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.527724 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.546885 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:49.553715 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.586654 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.590507 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:49.609068 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.609364 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.633832 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.636906 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:49.664285 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:49.693768 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.770511 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.778277 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:49.794385 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.795460 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.903496 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:49.942863 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:49.947799 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:49.983088 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.009701 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:50.016381 13559 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.016520 13559 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:50.017122 13559 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:50.017527 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
W20240503 01:59:50.017685 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:50.017740 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 01:59:50.018394 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:50.018471 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
W20240503 01:59:50.018584 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:50.018628 13559 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:50.081846 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:50.107829 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:50.235586 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.248843 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.250761 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.289783 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 01:59:50.300384 12687 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.300539 12687 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:50.301091 12687 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:50.301484 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 01:59:50.301649 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 01:59:50.302186 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:50.302366 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:50.302446 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:50.302570 12687 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:50.309016 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:50.319216 13404 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 01:59:50.319406 13404 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:50.319990 13404 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:50.320472 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:50.320647 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 01:59:50.321153 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.321234 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:50.321295 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:50.321439 13404 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:50.374117 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:50.388623 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.412442 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.512558 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:50.541755 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.561502 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:50.562407 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:50.648228 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.670130 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.704808 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.722841 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:50.744486 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:50.767081 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.801208 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.805568 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.830389 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.876673 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.883361 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.891953 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 01:59:50.897537 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:50.914021 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:50.919534 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 01:59:50.936333 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:50.940408 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:51.000484 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.006960 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:51.058746 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.083417 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.092530 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.122989 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:51.133352 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.210582 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.223933 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.252148 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 01:59:51.306247 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.330513 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:51.402705 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.434428 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.437849 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.439345 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 01:59:51.439913 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.440022 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:51.519779 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.521044 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.583914 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:51.611879 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.719048 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:51.728730 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:51.745858 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.750115 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.800935 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.819483 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:51.821588 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.829748 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.885710 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.952852 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.960806 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.966881 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:51.971143 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:51.988590 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.025712 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:52.149616 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.157680 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.214478 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.226233 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:52.248684 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.263759 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:52.282084 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.295774 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.300041 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.359236 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.381985 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:52.477811 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 01:59:52.485405 13218 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.485553 13218 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
W20240503 01:59:52.485833 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 01:59:52.486063 13218 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:52.486413 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:52.486572 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 01:59:52.486928 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:52.487046 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:52.487102 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:52.487227 13218 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:52.562144 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.576467 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:52.625700 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.628871 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.717455 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.747146 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:52.765900 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:52.779722 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.814656 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:52.868340 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:52.906500 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.970975 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:52.996712 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:53.062037 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:53.135360 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:53.186075 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.192279 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:53.249534 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.268277 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:53.317350 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:53.327394 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:53.330950 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.394466 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:53.412565 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.435776 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:53.440785 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.450196 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.456727 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:53.465204 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.476475 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.509753 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:53.549433 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.559973 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.560451 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:53.616946 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.630013 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.679004 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.704728 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:53.820228 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:53.834362 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.841643 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.878193 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:53.904455 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:53.931034 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:53.939410 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:54.010092 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.014382 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.017966 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:54.070811 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.120839 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.164204 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.216972 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.248458 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.293246 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.293617 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:54.328356 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.371227 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.382529 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.404603 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:54.421056 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:54.437235 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.466542 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.481114 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:54.488838 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.508163 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.588289 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.645138 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.656615 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.698524 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.739818 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.789865 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:54.810920 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:54.821612 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.834129 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.869931 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 01:59:54.880914 14000 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.881064 14000 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 01:59:54.881548 14000 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 01:59:54.881923 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 01:59:54.882083 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 01:59:54.882578 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:54.882681 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:54.882738 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 01:59:54.882843 14000 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:54.896433 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:54.969715 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:54.995380 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:55.024552 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.085183 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.090543 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 01:59:55.158303 13809 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.158488 13809 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:55.159016 13809 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:55.159715 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 01:59:55.159934 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 01:59:55.160429 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 01:59:55.160622 7570 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:55.160796 13809 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:55.160811 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:55.164412 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.170740 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.226231 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.319986 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.336369 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:55.364179 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.416399 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:55.420411 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:55.501065 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.530480 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.571008 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.599252 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 01:59:55.627214 14190 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 01:59:55.627377 14190 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 01:59:55.627884 14190 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 01:59:55.628332 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 01:59:55.628482 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 01:59:55.629336 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.629464 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 01:59:55.629514 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 01:59:55.629730 14190 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 01:59:55.641628 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.676239 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.816951 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.854887 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.864715 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:55.869549 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.873387 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.876906 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:55.901540 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240503 01:59:55.922524 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:55.929913 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.931610 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 01:59:55.936190 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:55.963133 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:55.982952 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:55.985129 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.055393 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.056211 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.083515 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.136860 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.147332 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.195328 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.204692 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:56.239192 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.284358 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 01:59:56.368942 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.388949 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.398375 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.437934 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.455583 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:56.455857 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 01:59:56.456876 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:56.459951 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.469312 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:56.470412 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.485162 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.489300 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.514730 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.542152 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.570770 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.592653 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.627148 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:56.736749 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.790251 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.801512 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.856132 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:56.860375 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:56.901512 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.936110 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:56.938153 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:56.986703 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.016499 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:57.024139 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.034193 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.041030 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:57.084314 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.162853 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.261250 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.312595 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.322990 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.323580 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.363327 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.371258 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:57.427227 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.516690 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.518954 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:57.547461 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.553756 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.582127 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:57.586699 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:57.652742 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.702872 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.757265 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.779697 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:57.860167 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 01:59:57.860963 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:57.883045 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.890673 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:57.901815 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:57.937965 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:57.949015 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:57.966470 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:57.996919 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:58.011497 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:58.025923 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.117062 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:58.182981 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.204039 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:58.224957 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.365870 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:58.377256 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.410791 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:58.418287 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.452558 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:58.480643 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:58.510607 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:58.512396 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:58.520486 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.524098 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.535413 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:58.592684 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:58.595749 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.621600 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.639191 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.681597 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.761526 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:58.762012 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.771589 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.778888 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.868749 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.881579 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:58.919643 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:58.924063 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:58.935189 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.014156 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.021154 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:59.032744 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.050594 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.088685 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.088832 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.093809 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.102113 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.154536 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:59.189972 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.191071 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.195207 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.292775 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.310030 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:59.332552 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:59.428624 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.475270 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.512853 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.560271 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 01:59:59.590713 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.614305 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.627175 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.649760 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 01:59:59.691303 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.716332 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:59.784063 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.869367 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.896917 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 01:59:59.908993 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 01:59:59.921213 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:59.971627 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.027433 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.042165 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.047770 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.096184 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.122001 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.163093 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.164815 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:00.194031 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.223039 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.298192 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:00.358639 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.365861 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:00.468809 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:00.500974 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.510921 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.550357 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.578114 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.618048 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:00.618876 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:00.656764 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.689440 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.742784 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.792491 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:00.894801 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:00.904904 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240503 02:00:00.905616 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.909519 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.929122 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:00.930362 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:00.955483 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:00:00.972134 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:00.982340 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.004865 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.019214 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:01.042045 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:01.051388 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.106072 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.153544 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.182348 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.228565 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.230494 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.257018 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.264277 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.270927 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.285961 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.320524 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:00:01.321856 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.345121 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:01.358270 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:01.374421 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.417552 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:01.423898 14624 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.424041 14624 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:01.424499 14624 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:01.424875 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:00:01.425058 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:00:01.425648 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:01.425714 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
I20240503 02:00:01.425869 14624 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:01.425885 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.427285 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.469022 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:01.484561 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.495357 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:01.512253 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:01.527297 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:01.566486 14687 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.566637 14687 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:01.567107 14687 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:01.567775 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:01.567922 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:01.568181 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.568260 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:01.568310 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:01.568442 14687 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:01.580672 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.627485 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.650285 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.746707 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.798496 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.809329 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:01.817842 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.860795 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:01.887392 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:01.946516 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.973397 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:01.990959 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:02.094974 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.099710 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.127188 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:02.136113 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:02.197465 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.236310 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:02.259083 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:02.320712 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.399088 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.412873 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:02.446199 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.449488 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.465248 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.470963 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.483665 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:02.588637 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.590448 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:02.601435 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:02.626461 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:02.656356 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:02.692289 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:02.712916 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:02.746971 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:02.832638 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:02.841320 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:02.841797 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:02.869553 14995 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:00:02.869709 14995 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:02.870254 14995 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:02.870725 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:00:02.870894 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:00:02.871603 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:02.872417 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:02.872498 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:00:02.872664 14995 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:02.982054 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:03.076297 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:03.121659 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.159325 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.180509 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.226559 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.233777 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:03.238137 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:03.245462 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:03.250869 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:03.367596 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.368685 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.383117 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:03.434859 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.543972 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.559226 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.579517 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.599418 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.635315 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.636579 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.647562 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:03.655596 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:03.656523 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:03.682431 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.699841 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:03.738656 13811 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.738823 13811 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:03.739521 13811 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:03.739835 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:00:03.740001 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:00:03.740674 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:03.740763 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:03.740845 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:03.741070 13811 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:03.741170 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.797346 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.802139 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.808491 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.812748 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:03.816349 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.821120 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:03.836882 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:03.898097 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.981009 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:03.996673 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.006704 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.057024 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.096252 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.109733 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.151814 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:04.155874 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.157023 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:04.164748 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.201326 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:04.258123 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.291013 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.291175 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:04.340698 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.401294 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.423835 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.463507 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.504163 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.574745 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:04.646791 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.647287 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.696938 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:04.760159 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:04.791803 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:04.848199 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.865893 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.877904 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.941953 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:04.946530 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:04.979390 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.002604 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.064222 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:05.178962 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:05.205785 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.279515 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.284238 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:05.341382 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.401275 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:05.413961 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:05.416646 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.418169 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.459692 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.476408 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:05.482326 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:05.485659 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:05.592851 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.646039 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:05.648742 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.660527 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.662098 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:05.752421 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.817287 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:05.848945 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:05.939270 15539 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.939445 15539 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:05.940032 15539 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:05.940500 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:05.940672 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:05.941191 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:05.941321 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:05.941398 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:05.941540 15539 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:05.944819 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:00:05.959271 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:00:05.967618 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.970321 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:05.979895 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:05.987159 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:05.987222 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.003583 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:06.012507 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.018956 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:06.079521 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.096102 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:06.107021 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:06.186566 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.234894 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.256115 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.266371 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.267012 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.298580 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.367771 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:06.375733 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:06.410508 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:06.416188 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:06.436944 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:00:06.471470 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.480602 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.536520 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:00:06.537284 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:06.568380 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.627111 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:06.635021 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.636937 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.659024 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.710155 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:06.711119 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.728371 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.762931 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:06.787659 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.839128 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:06.876921 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:06.979131 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.005823 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.013206 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.014292 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:07.051795 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.069617 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.088268 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:07.113543 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:07.125018 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.216357 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.231248 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.232174 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.239631 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.270066 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.319936 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:07.399926 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.526372 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.539500 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:07.608405 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:07.644497 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:07.650084 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.716221 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:07.828474 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:07.855734 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.858376 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:07.862905 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.869252 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.909649 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.931211 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:07.941664 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:07.989810 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:08.023152 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.083494 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:08.102412 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:08.182310 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:08.189606 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.198009 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.213282 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:08.225229 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:08.246230 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.341408 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.343636 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:08.365056 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:08.369154 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:08.461627 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.475152 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.486137 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.493515 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:08.495136 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:08.507426 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.507799 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.515429 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:08.581051 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.693325 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.706331 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:08.717268 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:08.734144 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.761147 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.792876 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:08.818853 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:08.855273 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.899216 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.920715 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:08.992476 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.040966 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.064067 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.066745 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.083914 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:09.088474 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.102197 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.107770 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:09.124049 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.189944 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.242233 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.244518 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:09.313544 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.331000 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.370924 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.393839 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.399612 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:09.438316 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.450085 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.495823 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.529816 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.560617 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:09.610958 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:09.638267 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.708655 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.723505 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.756513 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.776063 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:09.794638 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.832315 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:09.896236 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:09.964867 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:10.017333 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:10.054493 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:10.085336 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:10.109225 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.170892 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.244344 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.341084 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.360388 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.370941 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:10.380389 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:10.388691 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.414470 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.447865 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.468986 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:10.495765 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:10.513535 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:10.523610 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:10.637403 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:10.647264 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.692128 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.812316 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:10.839699 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.889717 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.896998 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:10.921612 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:10.924796 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:10.947270 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:10.965204 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.009364 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.015285 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 195 similar messages]
W20240503 02:00:11.018359 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:11.031867 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:00:11.046610 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.048879 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.063947 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:11.082594 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.092689 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.129705 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.186805 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:11.226260 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.254680 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.277652 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.307857 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.325898 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.340698 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.349951 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:11.353515 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.370780 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:11.399734 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.405326 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.483057 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:00:11.486178 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.491477 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:11.513203 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.577893 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.599858 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.625172 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.628966 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.633734 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:00:11.678368 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.748356 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:11.764822 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:11.808512 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:11.850390 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.879292 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.897956 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:11.899232 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.926669 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:11.953289 16290 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.953409 16290 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:11.954077 16290 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:11.954205 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:00:11.954380 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:00:11.955606 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:11.955679 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
I20240503 02:00:11.955860 16290 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:11.956213 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:11.972221 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:12.015938 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.040380 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.080936 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.091590 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.143931 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.153674 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.228796 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.240638 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:12.253733 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:12.311093 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.317931 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:12.340600 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.437887 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:12.454932 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.459193 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.496805 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:12.604589 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:12.629240 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.640259 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:12.651666 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:12.656374 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.683897 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:12.694013 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.726142 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:00:12.816221 13811 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.816372 13811 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:12.816829 13811 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:12.817284 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:00:12.817448 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:00:12.817991 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.818238 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:12.818315 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:12.818459 13811 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:12.832278 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:12.867528 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:12.919693 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.992954 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:12.998791 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.074180 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:13.092669 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:13.204247 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:13.211488 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.222961 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.247660 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.296587 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.318426 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.329085 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:13.348430 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:13.408990 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:13.432104 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.477372 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.488456 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:13.504436 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.532459 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.605302 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:13.615172 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.650303 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.659622 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.679023 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:13.697330 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.705335 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.722487 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.737696 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.744460 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.753257 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.774564 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.835901 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.841876 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:13.853374 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:13.887333 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.922072 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:13.947217 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:13.977511 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:14.081487 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.084338 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.107795 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.138337 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.158026 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:14.162345 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.243477 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.245405 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.265029 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:00:14.265650 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:14.310967 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.361853 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:14.379339 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:14.416380 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.423705 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.435217 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.552592 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:14.553748 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.556970 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:14.562184 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.575523 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.606413 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.708062 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.710336 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.741954 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.750556 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.761909 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:14.800191 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:14.814393 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:14.883960 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:14.904304 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:14.931056 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:14.953132 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:15.018414 13811 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.018570 13811 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:15.019523 13811 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:15.019902 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:00:15.020061 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:00:15.020608 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.020879 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:15.020953 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:15.021423 13811 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:15.165642 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.177402 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:15.294191 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.297746 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.337705 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.341327 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:15.341653 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.352718 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:15.368242 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:15.384089 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:15.398074 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.450685 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.481850 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.581929 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.644480 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.728649 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.763759 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:15.765501 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.786618 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.825292 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:15.842243 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.852216 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:15.867725 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.879499 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:15.902011 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:15.914808 16986 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:00:15.914952 16986 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:15.915437 16986 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:15.915848 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:00:15.916024 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:00:15.916457 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:15.916637 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:15.916687 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:00:15.916818 16986 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:15.921574 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:15.925532 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:15.937309 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:15.976222 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.015198 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.046756 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 194 similar messages]
W20240503 02:00:16.072580 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240503 02:00:16.073275 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.079942 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.095625 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.133157 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:16.157742 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.160295 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.192426 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.200888 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:16.217195 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:16.281141 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:16.281584 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:16.303278 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.330253 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:16.359611 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.455197 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:16.485971 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:16.486850 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.492009 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.502188 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.543746 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:00:16.611773 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:16.658768 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:00:16.666749 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:16.697554 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.772017 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.776939 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.806170 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:16.854305 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.862054 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:16.883028 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.894481 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.902494 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:16.903949 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:16.906078 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:17.044740 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.068670 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:17.136647 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.161868 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:17.172052 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:17.184576 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:17.194007 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:00:17.223699 13811 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:17.223855 13811 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:17.224308 13811 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:17.224772 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:00:17.224964 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:00:17.225536 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:17.225648 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:17.225713 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:17.225865 13811 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:17.276211 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:17.305795 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:17.328563 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.345119 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.394486 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:17.409468 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:17.420852 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.432428 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.559185 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:17.629922 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:17.641117 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:17.688426 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.691232 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:17.722098 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:17.790639 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:17.793711 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:17.805413 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:17.842567 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:17.925038 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:17.944048 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:18.016754 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.019161 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.106792 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:18.150168 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.158336 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.158418 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:18.166488 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.231078 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:18.236635 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:18.311899 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:18.358619 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.402437 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.403070 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.442327 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.455026 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.459923 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:18.462035 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:18.462674 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:18.510681 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.616943 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:00:18.626077 16986 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:00:18.626220 16986 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:18.626741 16986 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:18.627137 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:00:18.627285 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:00:18.627738 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.627869 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:18.627951 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:00:18.628078 16986 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:18.632790 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.636046 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.640981 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.681586 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:18.718135 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.742265 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.768321 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.777964 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:18.778671 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.810343 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.830446 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.830747 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.841594 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:18.842547 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:18.868414 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.907589 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:18.961408 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:18.989497 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.017663 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.047986 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.188059 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.192279 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.207015 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:19.264863 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:19.303418 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.321913 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.410236 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:19.432284 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:19.438688 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:19.478392 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.578088 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.597671 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.619287 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.671450 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:19.681195 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.690124 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:19.729458 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.786696 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:19.805241 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.816852 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.842221 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.868582 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.879184 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:19.891830 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:19.917472 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:19.963099 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:20.012251 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.041064 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.093798 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.104261 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.130671 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.197131 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.206998 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:20.257537 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:20.277354 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.448303 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.470728 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.508174 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:20.519881 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:20.623816 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.684206 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.717068 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.755834 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:20.813187 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.827165 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:20.866873 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.875640 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:20.885605 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.885679 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.923290 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.931226 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:20.944171 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:20.966361 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.972226 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:20.979348 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:20.986588 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.009166 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.047753 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 187 similar messages]
W20240503 02:00:21.077906 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:00:21.082384 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.109441 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.111253 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.167227 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.170531 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:21.178221 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.197269 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.256031 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:21.261648 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.262507 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.292991 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.319406 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.319947 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.343195 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.358299 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.371295 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.391723 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.419940 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.457181 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:21.473949 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:21.484890 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.624140 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 02:00:21.627630 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:21.711063 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.712744 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:00:21.724364 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.789026 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:21.902454 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:21.903295 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:21.910471 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:21.993358 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:22.003265 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:22.003335 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.009385 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.094594 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.094671 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.141539 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.165738 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:22.215256 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.225695 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.253875 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:22.300472 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.327750 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.352770 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:22.408685 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.426698 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.443944 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:22.490598 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.513173 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:22.547888 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.568113 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.572552 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.577255 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.592411 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.607170 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:22.607420 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:22.621994 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:22.806663 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:22.837209 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:22.944458 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:22.987681 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:23.005168 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:23.168859 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.173259 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:23.184964 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.187331 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.211519 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.301800 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:23.339095 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.371220 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:23.382094 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:23.425567 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.429502 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.466037 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.478092 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:23.516721 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.548807 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:23.559993 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.566067 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.588384 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.595497 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.631788 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:23.644523 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.667580 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.667997 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.678679 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.682375 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.702361 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.725483 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:23.728134 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:23.733963 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.805073 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.845583 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.857379 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:23.882799 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:23.945057 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:23.955394 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:24.003072 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.007642 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.113919 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:24.119381 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:24.178419 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.251495 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.315632 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.339902 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:24.371600 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:24.408654 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.459484 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.560513 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.577551 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:24.585875 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.593343 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:24.666796 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.687218 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:24.725908 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.752246 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.778749 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.779584 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:24.788030 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:24.820097 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.906047 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.909834 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.924712 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:24.940675 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.955253 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:24.955763 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:24.986721 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:24.989102 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.010890 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.019666 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:25.115149 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.241444 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.260885 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.272320 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:25.296159 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:25.375018 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.378677 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:25.408573 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.515935 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.535828 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.541493 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:25.651250 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:25.685516 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.688900 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.726276 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.743306 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.773649 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.781872 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.791833 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.826953 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.843468 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:25.886986 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:25.894812 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:25.939102 17653 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:00:25.939263 17653 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:25.939877 17653 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:25.940279 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:00:25.940446 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:00:25.940738 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.940865 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:25.940944 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:00:25.941068 17653 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:25.949331 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.958117 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:25.978260 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:25.980188 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:00:25.984803 17622 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.984933 17622 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:25.985491 17622 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:25.985814 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:25.985970 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:25.986394 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.986450 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:25.986553 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:25.986625 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:25.986737 17622 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:26.037458 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.053524 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240503 02:00:26.064662 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.079038 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:00:26.110687 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:26.124037 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.165805 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:26.166237 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.181747 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.187829 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:26.196321 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.212056 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.259400 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:26.272558 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.340018 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:26.342440 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.375895 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:26.402115 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:26.501861 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:26.529073 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:26.568462 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.577704 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.654472 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.695369 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 02:00:26.730283 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:26.743553 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:26.761500 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:00:26.828161 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.833503 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:26.867437 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:26.953475 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:26.967104 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.014746 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:27.053009 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.062089 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.075219 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.127100 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.203042 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.255578 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.261286 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:27.300585 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.337234 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.341792 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.356607 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:27.377447 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:27.406175 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.413033 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.444864 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.455022 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:27.480280 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.510800 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.541486 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.559806 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.574038 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.586915 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.591907 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:27.702693 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.770174 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:27.789819 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:27.817313 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.824378 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:27.896131 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:27.935184 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:27.943636 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:27.952613 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:28.067675 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.150609 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.164711 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:28.206676 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:28.215922 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:28.227432 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.290378 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:28.297904 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.349843 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:28.369946 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.375694 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.429939 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.445183 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:28.447515 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.464531 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.495157 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:28.510895 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.522948 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.557144 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.567808 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.593087 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:28.620072 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.637228 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:28.650542 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.672772 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:28.697217 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.724805 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.770856 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.842522 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.896697 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:28.947003 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:28.978539 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:28.981549 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:29.058357 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.114683 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:29.146239 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.200572 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.249375 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.250931 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.256961 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:29.282469 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.361434 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:29.394186 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.415306 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.477353 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.501256 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.525930 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:29.545519 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:29.601676 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:29.617564 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.635461 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:29.640056 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:29.738386 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.741446 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.741935 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.835754 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.863624 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:29.907989 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.911864 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.935478 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:29.945871 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:29.952870 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.954141 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:29.957258 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:30.084551 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.101914 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.104080 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.139295 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:30.173871 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.190547 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.203066 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:30.346956 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.397894 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.400457 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.432376 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.458482 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.489390 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:30.575625 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:30.611006 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.659677 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:30.716040 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.719017 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.747200 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.815176 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.822307 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:30.837265 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.879223 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.890512 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.897912 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:30.918431 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.933189 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.936590 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.938243 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:30.944521 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:30.952608 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.079907 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240503 02:00:31.080626 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:31.131263 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:00:31.133687 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:31.145088 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.203187 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:31.225960 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:31.226284 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:00:31.261324 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.285506 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:31.366592 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.379678 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.408776 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.436786 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:31.438748 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:31.524717 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:31.534909 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.621345 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.634747 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.639968 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:31.646986 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:31.663342 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.738039 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:00:31.762567 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:31.831365 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:31.838124 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.875831 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:31.875926 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:31.879994 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.905186 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:31.969630 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:31.983183 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:32.027359 18921 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:32.027511 18921 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:32.028172 18921 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:32.028625 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:00:32.028784 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:00:32.029438 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:32.029527 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:32.029615 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:32.029778 18921 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:32.067786 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:32.089273 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:32.110855 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:32.123004 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.139744 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.161972 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.242575 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.264331 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:32.266532 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:32.364374 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:32.370743 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.416751 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.453305 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:32.482944 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.483917 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.495503 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:32.507614 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:32.520486 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.545831 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.585986 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:32.595499 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:32.721843 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:32.731451 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:32.758955 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:32.765678 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:32.833887 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:32.909585 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:32.982390 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.080794 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.090281 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.097997 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:33.137538 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:33.150274 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:33.192243 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:33.192699 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:33.195997 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:33.260826 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:33.266677 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.280037 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.290088 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.291899 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.350282 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.386211 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.423339 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.430910 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.457815 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:33.464222 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.490753 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.540567 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.553681 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:33.571094 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:33.583731 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.587862 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:33.625928 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:33.681704 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.757552 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.781687 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.790969 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:00:33.815866 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.887521 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.896780 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:33.921962 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:33.994664 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.029855 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.070027 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.082603 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:34.084002 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.111196 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:34.150626 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.153259 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:34.200196 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:34.202347 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.327761 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:34.336068 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.352802 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:34.363607 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:34.424283 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.437922 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.561534 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:34.572475 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:34.588933 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.597962 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.641062 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:34.659906 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:34.663817 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.668071 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:34.738574 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:34.790097 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:34.827802 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.834303 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:34.861601 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:34.883340 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.006356 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.014204 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:35.063869 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.066751 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.107951 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:35.156939 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:35.228377 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.232051 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.249526 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.258949 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:35.307579 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.360882 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.379204 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.418979 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.441368 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.517099 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:35.541046 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:35.562963 18612 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.563115 18612 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:35.563637 18612 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:35.563979 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:35.564121 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:35.564529 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:35.564882 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:35.564934 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:35.565048 18612 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:35.566411 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.628882 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.636000 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.703372 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:35.705883 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:35.715279 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.723793 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.796010 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.802929 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:35.892243 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:35.910521 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:35.939057 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:35.959399 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.993906 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:35.998754 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.000734 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.009622 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:36.032832 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.048554 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.061764 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.113652 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:00:36.168887 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:00:36.172927 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:36.198401 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.214265 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.236649 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.299018 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:00:36.302814 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.325973 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:36.354154 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:36.363725 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.393316 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.408397 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.501407 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.535820 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.549664 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:36.593264 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.604461 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.634491 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.671109 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.714496 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.733026 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:36.758505 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:36.774050 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.784622 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:36.821327 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:00:36.886651 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:36.890231 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.896793 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:36.941680 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:36.976547 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:36.984515 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.085250 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.116710 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:37.149048 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.203208 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.245142 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.280814 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.291219 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.294590 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:37.326707 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:37.348678 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:37.351202 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.352593 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:37.402276 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.481909 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.484855 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.490368 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.557389 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:37.644059 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:37.749905 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:37.840518 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.841240 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:37.845805 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:37.878233 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.884434 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:37.952000 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:37.956378 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:37.977751 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:37.993232 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.045898 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.053489 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.138542 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.168077 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:38.292397 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.297278 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.314893 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:38.349664 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:38.377353 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:38.390785 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:38.402475 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.447497 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.457962 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.485483 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:38.506115 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.510759 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.524958 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.528348 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:38.667055 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.673144 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.674643 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.752846 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:38.753383 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:00:38.756673 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.765537 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.773087 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:38.827953 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.875825 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:38.882388 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:38.926146 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.959633 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:38.981882 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:38.992257 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:38.998811 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.011490 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.035948 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.097409 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.163225 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.169647 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.227109 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:39.242019 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.246163 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.262687 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.293128 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:39.348276 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.415709 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:39.458479 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.465580 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.467707 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.480317 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.496520 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:39.570096 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:39.679224 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:39.698693 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.702558 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.720090 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.745157 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.777693 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:39.804978 20032 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.805111 20032 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:39.805534 20032 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:39.805912 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:39.806118 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:39.806638 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:39.806785 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:39.806834 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:39.806962 20032 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:39.832935 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:39.843927 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:39.888051 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.924930 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:39.930169 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:39.943836 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:40.010679 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:40.117309 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:40.128039 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:40.182089 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:40.221272 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.252565 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.256250 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:40.321152 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.352749 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.412945 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:40.462723 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:40.490679 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:40.516330 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.590528 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.663785 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:40.708331 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:40.716034 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.724470 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.769040 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:40.793731 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:40.818765 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.819610 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.840137 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:40.874758 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.912199 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.919979 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.984930 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:40.987907 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:41.048353 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:41.048460 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.062197 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.093660 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:41.099862 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:41.109833 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:41.122133 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240503 02:00:41.122749 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:41.131780 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.152302 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:00:41.172863 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.213804 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:41.234282 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:00:41.234891 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.291455 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:41.353339 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:41.357609 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.374716 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.441627 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:41.443939 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.451354 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.548899 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:41.553484 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:41.557839 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.558840 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:41.588747 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.680928 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.707737 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.729610 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.735796 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.828410 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:00:41.829110 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:41.881825 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:41.882890 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.907338 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.928614 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:41.947476 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:41.952775 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:41.952757 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:00:41.953454 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:42.065501 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:42.142812 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:42.176431 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.239343 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:42.266805 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.279103 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.310084 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:42.319717 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.394207 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:42.400477 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.406934 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:42.408594 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:42.477228 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:42.479094 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:42.525626 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:42.565488 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.662081 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:42.672255 19353 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:00:42.672380 19353 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:42.672912 19353 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:42.673300 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:00:42.673456 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:00:42.673924 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:42.674002 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:42.674085 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:00:42.674228 19353 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:42.683079 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.699821 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:42.738802 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:42.738888 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:42.777774 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:42.799046 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:42.872557 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:42.925590 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:42.987826 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.034040 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:43.099346 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.141865 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:43.148823 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.172225 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.261368 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.266958 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:43.286559 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.297330 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.297959 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.323573 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.337383 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.361649 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.380033 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.410450 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.459160 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.466114 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:43.517220 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:43.540367 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.547075 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.556866 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.579043 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:43.603243 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.649698 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:00:43.669109 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:43.670181 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.692552 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.709959 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.795109 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.803987 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:43.829115 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.912056 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.944559 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.948535 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:43.965291 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:43.971083 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:43.971753 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.074069 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:44.086427 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.100242 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.129217 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.172032 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.191341 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:44.237430 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.241245 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.324627 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.328822 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.403861 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.410552 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.420847 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.480712 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:44.492859 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:44.574399 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.625013 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.637130 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.650130 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:44.739970 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:44.744066 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:44.775521 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.803360 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.912807 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:44.920092 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:44.948678 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:44.961428 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:45.019371 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.037858 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.051854 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:45.096870 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.108335 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.121291 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.156059 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.176183 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:45.204365 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.236832 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.290493 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.300372 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.345934 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:45.370021 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.439229 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:45.454527 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:45.502043 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.563295 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.657830 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.674189 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.686206 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.696311 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:45.722684 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.743499 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.765844 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.782250 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.835536 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:45.883926 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.910072 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.914953 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:45.938148 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:45.958956 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:45.961633 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:46.024322 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.026758 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:46.046276 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:46.103379 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:46.115945 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.119846 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:46.132798 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:46.166402 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:46.204463 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:00:46.205245 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.233136 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:00:46.240540 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:00:46.265200 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:46.318642 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.428261 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:46.447032 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.458370 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.482720 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:46.516140 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.535251 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.538829 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:46.569665 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.570873 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.576123 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:46.633314 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.649463 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:46.684965 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:46.698886 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.699476 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.717155 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.737380 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:46.815148 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.831257 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:00:46.831951 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:46.971457 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:00:46.987891 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.024758 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:47.041384 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:00:47.042168 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:47.056275 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.089570 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:47.106510 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:47.107650 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.130359 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:47.140679 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.154793 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.238615 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.320956 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:47.339743 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.385728 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:47.457932 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.488870 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:47.516755 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.540922 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.563324 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:47.578042 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:47.579136 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:47.595492 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:47.617028 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.653182 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.718454 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.742228 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.823341 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:47.916746 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:47.919700 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:47.969798 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.032658 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:48.033139 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:48.046343 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.100808 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.139489 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.151481 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:48.154456 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.159521 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.162451 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.164785 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:48.203917 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:48.291018 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.309576 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.336078 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.374702 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.394192 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.400027 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.409070 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:48.418673 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.469838 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.474645 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:48.522833 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.526450 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:00:48.537425 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.618736 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:48.650652 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:48.663746 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.770377 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.794468 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.851167 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.892091 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:48.897081 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.900696 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.917642 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:48.938771 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:48.958899 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:48.961733 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.004793 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:49.049775 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.099608 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.126382 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.135222 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.138378 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.154544 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.162318 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:49.196640 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.203720 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.242848 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.298197 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.344308 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.420028 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:49.420076 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:49.509290 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.565606 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:49.596848 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.602715 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.648025 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:49.658339 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.695317 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:49.746392 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.777992 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:49.866011 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:49.900034 21203 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:49.900206 21203 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:00:49.900861 21203 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:00:49.901340 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:00:49.901501 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:00:49.901890 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.901996 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:00:49.902081 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:00:49.902207 21203 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:49.911710 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:49.936270 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:49.960700 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:49.967492 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:49.969986 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:49.976042 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:50.088150 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.091956 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.097143 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.153343 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.230809 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:50.266561 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.268705 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.339457 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.397431 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:50.440124 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:50.522540 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.536036 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.556713 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.586573 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.597218 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:50.621011 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.657827 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.667290 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:50.701771 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.712034 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.718998 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.757694 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:50.796016 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.856156 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.902498 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:50.916751 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:50.923713 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:50.941598 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.007440 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.008168 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:51.018844 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.067050 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:51.089228 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:51.103246 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.115577 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.144400 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:00:51.208401 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:00:51.209072 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.225159 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.279968 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:00:51.285519 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.305685 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:51.308792 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.323513 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.327368 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.381140 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.410732 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:51.425309 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:51.489349 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.501230 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.521823 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.564462 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.575126 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:51.587455 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.626411 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.644157 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.655428 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.692677 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.721976 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.744046 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:51.775951 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.892190 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:51.898813 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:51.905974 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:51.933784 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:51.958176 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:00:51.958803 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.056257 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:52.085968 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.089131 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:52.140622 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.179937 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.192097 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:52.209502 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:52.356110 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.374240 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.390667 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:52.404944 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:52.465868 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:52.501299 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.509248 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.555833 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.617750 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.623684 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:52.647221 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:52.734882 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:52.744763 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.785039 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.824012 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:52.825543 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:52.880060 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:52.911672 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.012864 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.065920 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:53.066388 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.081921 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:53.116000 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.121235 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.127506 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.143779 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.144729 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.196197 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.202234 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.236101 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.263880 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.287812 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:53.337574 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:53.346364 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.357241 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.363250 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.462678 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.466069 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.476827 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:53.495043 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:53.533319 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.579942 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.581380 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.593902 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.661901 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.719856 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.726212 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.733973 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:53.745781 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:53.782255 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:00:53.804069 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:53.941927 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:53.968248 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.003176 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.024179 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.028788 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:54.030968 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.054461 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.093677 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:54.146230 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:54.147111 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.172135 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.172142 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.204334 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.227942 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.230667 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.247251 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.254000 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.271498 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:54.447327 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:54.505995 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:54.541312 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.554298 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:54.609287 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.656126 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.667573 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:54.668859 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.670114 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:54.671407 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.730954 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:54.778277 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.836982 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:54.869975 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:54.938516 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:55.027956 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.062788 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:55.125591 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:55.198969 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.209401 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.240973 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:55.272663 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.293479 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.313702 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.360656 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:55.377117 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.381332 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.392706 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.464733 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.494194 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.508520 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.510349 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.554996 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:55.588369 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.650130 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:55.660641 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.672384 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:55.682978 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.705536 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.784247 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.812781 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:55.844488 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:55.849575 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:55.934669 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:55.994033 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.024891 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.034435 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:56.055346 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.065239 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:56.081128 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:56.084290 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.130101 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.132333 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:56.145888 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.154028 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:56.165896 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:56.183048 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.187237 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.222395 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.267832 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:00:56.278965 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:00:56.280687 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 02:00:56.281322 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.357113 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.445638 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:56.498373 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.498946 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.543233 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.607558 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.627728 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:56.633024 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.662463 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:56.662581 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.680759 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:56.717505 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.753057 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:56.761617 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.813257 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.903851 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.936317 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:00:56.952658 21700 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:56.952782 21700 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:00:56.953320 21700 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:00:56.953720 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:00:56.953894 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:00:56.954453 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:00:56.954521 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:00:56.954607 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:00:56.954692 21700 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:56.955053 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:56.985214 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:00:57.006827 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.015609 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:57.017316 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.040715 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.089183 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:00:57.113924 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.121066 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:57.121745 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:57.130587 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:57.157964 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.242966 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:57.285674 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.318409 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:57.347198 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.461760 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:57.477427 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.526899 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:57.533653 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.577989 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:57.664443 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:57.719506 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:57.736217 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.756898 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.847047 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.863663 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.864131 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:57.867511 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.979794 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:57.991490 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.053207 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:58.069983 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.072077 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.073133 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.095216 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.114707 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:58.129173 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:58.187207 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.240820 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.286564 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:58.298777 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.306207 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.322062 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.330116 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.334764 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.423141 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.442348 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.459918 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.482399 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.521288 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:58.522928 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.527963 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.601418 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.609098 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:58.610136 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:00:58.624857 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.628931 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:58.758574 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.807216 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:58.809552 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.838379 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.862987 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.868052 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.890781 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:58.946158 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:58.948537 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.035826 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.053431 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.059837 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:59.073279 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.091056 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.115921 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.123411 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:59.128499 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:59.182406 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:59.248287 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.265018 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.326947 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.340605 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.370945 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.372009 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.463011 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.517086 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:00:59.552989 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:59.558708 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.561969 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:59.581348 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.683625 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:00:59.732481 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:00:59.751441 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.757550 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.823961 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:00:59.949230 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:00:59.952872 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:00:59.959679 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:00:59.960963 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.089107 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:00.096720 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:00.108280 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.189298 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.220357 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.223346 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:00.243054 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.346356 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.387388 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.400290 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.418572 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:00.419142 22621 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:00.419277 22621 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:00.419838 22621 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:00.420146 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:00.420315 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:00.420804 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.421195 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:00.421252 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:00.421356 22621 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:00.495261 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:00.503652 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.517666 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:00.518959 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.521345 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:00.582945 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.597383 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.600659 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:00.621023 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.690739 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.707919 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.810150 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:00.837175 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.893366 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:00.912904 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.922385 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:00.931918 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:00.934307 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:00.954578 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.035938 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:01.065927 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:01.080237 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:01.087129 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:01.122886 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:01.127259 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.137470 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.142613 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.174099 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.185199 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.257431 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:01:01.283789 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:01:01.303973 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:01.330408 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.341516 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:01:01.351358 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.394819 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.422065 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:01.423362 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.427603 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.451382 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.451412 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.473052 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:01.540997 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.564533 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:01.652701 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.682628 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.693630 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.738607 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.803642 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.842732 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:01.864554 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.909564 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.915516 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:01.934255 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:01.954535 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:01.971949 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:02.037518 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:01:02.042685 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:02.072081 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.083695 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:02.085794 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.138887 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:02.139525 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:02.174993 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.208639 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.332020 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:02.335433 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:02.390878 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:02.416028 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.454224 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:02.510987 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:02.533252 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:02.574939 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:02.586441 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:02.603695 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.647394 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.652360 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:02.738605 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.836417 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:02.882083 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:02.900365 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.917068 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:02.920423 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:02.954337 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:02.973657 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.004583 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.048651 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.067847 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.082116 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.095348 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:03.097486 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.115903 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.164758 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:01:03.176538 23054 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:03.176709 23054 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:03.177222 23054 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:03.177783 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:03.177953 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:03.178195 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.178756 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:03.178831 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:03.178977 23054 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:03.199246 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.220727 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.306720 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:03.371498 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.419983 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.465487 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.501014 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:03.502970 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.543579 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.569470 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.578904 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:03.590284 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.603305 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.610821 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:03.617534 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.654839 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:03.680181 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.691761 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.727178 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.815968 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.853699 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:03.881029 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.886133 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:01:03.899734 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:03.924427 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.001605 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.018769 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:04.027424 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.048492 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.052124 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.118158 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:04.119791 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.161780 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.180115 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.180253 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:04.260922 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:04.326345 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.390020 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.405884 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.444216 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.467252 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.474370 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.549021 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.594084 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:04.600349 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:04.623061 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.628830 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:04.643900 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:04.672957 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:04.708140 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:04.716264 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:04.765753 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:04.901899 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.025537 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.050184 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.075338 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.078027 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.092379 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:05.097088 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.136916 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:05.172103 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.188199 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.204628 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.205890 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:05.214377 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:05.267908 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.311600 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.342836 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.461853 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.480419 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:05.524948 22929 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:05.525115 22929 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:05.525733 22929 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:05.526185 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:01:05.526374 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:01:05.526979 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.527102 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:01:05.527159 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:05.527276 22929 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:05.542768 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.573629 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:05.624245 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.638092 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:05.638773 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.673807 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.685834 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.696322 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.744347 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:05.804613 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.809038 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:05.827975 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.836961 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.880542 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:05.886514 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:05.913666 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:05.916805 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.003170 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.080490 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.101851 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:06.108963 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:06.110314 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.122687 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:06.141332 23455 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:06.141461 23455 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:06.141876 23455 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:06.142246 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:06.142401 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:06.143049 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.143141 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:06.143205 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:06.143337 23455 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:06.151573 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:06.163173 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:06.180052 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.211017 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.252235 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.299100 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.305451 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:01:06.328403 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:06.347450 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 02:01:06.370937 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.415714 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:01:06.425992 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.486526 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.512859 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.546737 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.556097 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:06.570776 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:06.573856 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:06.579114 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.603997 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.691434 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.703866 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.732378 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.752096 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.817529 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.858047 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.863092 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:06.909081 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:06.911295 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:06.987412 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:07.038733 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:01:07.039883 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.068883 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:07.069730 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:07.097360 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:07.135999 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.198416 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:07.205646 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:07.251410 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.278781 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.298965 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:07.413512 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:07.423738 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:07.486388 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.508333 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:01:07.525720 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:07.580715 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:07.596074 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:07.670945 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:07.679626 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:07.730962 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:07.765525 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.782277 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:07.799170 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.836577 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.851939 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:07.971741 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:07.999173 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.005821 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.017843 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.020591 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:08.046312 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:08.102090 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:08.147857 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:08.205179 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.304441 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:08.313757 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:08.350633 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.386665 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.420475 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.472388 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:08.505596 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.508378 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.529925 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.548637 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:08.566061 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.566968 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.603915 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.607054 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.608994 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:08.612447 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.641412 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.644014 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:08.660516 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:08.679965 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.702841 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:08.712136 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:08.833071 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:08.914045 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.918421 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:08.920291 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:08.932653 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.003682 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.015223 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.075280 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.093444 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.132189 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:09.173813 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:09.183059 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.203181 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.213261 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:09.243350 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.287101 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:01:09.289362 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.294775 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.315640 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.330818 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.471055 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:09.481065 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:09.505576 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.515081 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.528787 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.570206 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:09.595718 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:09.633648 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.637656 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:09.645128 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.649066 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.757920 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:09.868592 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:09.893080 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:09.954627 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:09.991835 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:09.992206 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:10.070223 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.081455 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.085249 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.095464 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:10.134479 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.175837 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:10.237519 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.404543 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:10.408427 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.448716 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:10.485875 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.516069 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.528669 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.551074 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.567236 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.568790 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:10.619118 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.681550 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.685186 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.718498 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.772328 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:10.788916 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:10.800539 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.813537 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.841966 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:10.909862 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:10.938426 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.977188 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.995853 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:10.996398 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:11.012540 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.030608 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.047462 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:11.071174 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.122872 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.133839 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:11.197292 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.200592 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.207182 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:11.244381 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:11.258411 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.281891 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.317706 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240503 02:01:11.318354 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.396720 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.442076 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.460248 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:11.485781 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:01:11.490970 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:11.509879 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.524328 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.594074 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.625276 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.639763 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:11.693297 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.693940 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.744060 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:11.773938 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.783591 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:01:11.797427 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:11.891275 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:01:11.897830 24350 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:11.897960 24350 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
W20240503 02:01:11.898463 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:01:11.898532 24350 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:11.898882 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:01:11.899032 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:01:11.899436 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:11.899511 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:11.899626 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:11.899745 24350 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:11.909734 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.920181 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:11.921916 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:11.987125 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:12.003041 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.020473 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:12.043871 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:01:12.044523 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.057297 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.078241 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.111351 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.113901 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.286476 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:12.286778 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:12.369516 24319 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:12.369633 24319 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:12.370054 24319 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:12.370399 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:12.370563 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:12.371018 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.371372 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:12.371440 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:12.371604 24319 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:12.371853 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.426681 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:12.454161 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.482663 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.566857 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:12.597429 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:12.598196 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:12.626958 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.665938 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.667902 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:12.686295 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:12.710052 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:12.710110 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:12.899394 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:12.915881 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.017611 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.018231 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:13.073491 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.083235 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.092096 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.098155 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:13.133934 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.138620 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.150388 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.178052 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:13.199678 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.216248 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.275564 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:13.279278 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.302317 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.372136 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.399467 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:01:13.450752 24350 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:13.450882 24350 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:13.451274 24350 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:13.451696 7240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:01:13.451864 7240 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:01:13.452355 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:13.452446 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:13.452534 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:13.452682 24350 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:13.488564 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:13.501446 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.548272 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:13.574709 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.611819 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.622632 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.648272 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.657086 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.673494 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.728529 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.773960 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.798663 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:13.798764 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.807262 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:13.851485 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.876653 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.877033 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:13.914673 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:13.943171 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:13.960587 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:13.961594 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.055651 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.099045 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.120168 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.126175 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:14.208884 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.217196 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:14.236115 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.239281 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:14.261214 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.297993 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.396325 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:14.397328 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.413475 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.437424 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:01:14.456799 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.477350 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.500036 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.585638 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.617501 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:14.670398 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.673220 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:14.682619 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.717934 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:14.725721 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.751319 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:14.792776 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:14.796226 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:14.825240 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:14.924571 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:14.932361 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:15.023838 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.044057 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.114223 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.208523 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:15.214772 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.252262 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:15.257545 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:15.277222 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:15.348284 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.393028 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.491134 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.507005 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.530303 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.536499 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:15.539252 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.564389 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:15.568343 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.614040 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:15.656971 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:15.706676 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:15.732483 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.751770 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.762961 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.775701 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:15.824236 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:15.836258 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:15.861790 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:15.950304 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:15.961886 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.010680 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.033593 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.109422 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:16.158470 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:16.165971 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.169221 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.182358 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.190795 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:16.193439 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.218459 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.223302 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:16.328344 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.329922 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.330801 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.346064 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:01:16.396718 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:16.399713 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:16.433944 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.457546 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.463799 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.466492 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.481808 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.503746 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:01:16.504415 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.555631 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:16.639251 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:16.688100 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.690472 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.752727 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:16.845016 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:16.853765 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:16.859563 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.891120 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.908797 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:16.957152 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.017217 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:17.027722 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.088027 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:01:17.090104 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.098974 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.119166 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:01:17.165753 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:17.188731 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:17.205885 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:17.216280 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.276111 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:17.290539 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.311321 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:17.344518 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:17.356891 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.386428 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:17.437305 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:17.476361 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:17.589795 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.595142 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.596365 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:17.635028 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:17.705145 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.754307 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:17.806035 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:17.887741 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:17.889940 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:17.910108 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:17.930131 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:01:17.930760 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:17.956207 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.008633 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.047271 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:18.094034 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.104590 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.106348 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.108633 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.148797 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.173480 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:18.204340 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.208207 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:18.346084 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.389417 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.413628 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.462383 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:18.469728 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.482065 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.501153 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.536918 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.563748 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:18.609352 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.623454 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:18.624059 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.630898 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.641410 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.682312 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.727764 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.747562 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:18.750514 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.765797 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:18.794121 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.794709 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.824277 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.839516 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:18.895032 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:18.960456 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:18.969069 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.032428 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.073716 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:19.143465 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:19.156759 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.164165 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:19.173324 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.232493 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.253197 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.324456 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:19.354027 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:19.366747 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.391016 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.405371 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.414387 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:19.471428 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.590401 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:01:19.654289 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:19.660108 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:19.668061 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:19.729658 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.739482 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.746062 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.784512 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:19.823863 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:19.882802 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.894801 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:19.930380 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:19.990034 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.053194 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.077095 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.079602 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.104616 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.138720 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:20.152331 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:20.178059 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:20.234010 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:20.252084 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:20.294803 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.314245 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.344486 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.367769 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:20.409695 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.460966 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.462167 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.474573 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:20.490411 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.540364 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.588383 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.591900 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:20.601168 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:20.620146 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.698868 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:20.751951 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:20.773291 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:20.793664 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:20.902199 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.908038 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:20.946215 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:20.984459 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.018280 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:01:21.021167 25550 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:21.021279 25550 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:21.021807 25550 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:21.022162 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:01:21.022303 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:01:21.022918 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:21.023000 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:21.023072 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:21.023188 25550 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:21.053972 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:21.063009 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.087067 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.106092 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:21.109781 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.139278 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.171725 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.171814 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:21.190619 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:21.192931 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:21.222641 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:21.296676 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.343174 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.361701 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.363759 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:01:21.392802 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.403687 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:21.412364 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.418386 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.535655 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.548444 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:21.596122 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:01:21.663432 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.681165 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.703114 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:21.713269 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.737370 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:21.745196 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.783335 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:21.811949 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:21.852407 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.876919 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:21.931231 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:21.973512 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:21.987697 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:22.081081 25582 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:22.081215 25582 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:22.081812 25582 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:22.082124 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:01:22.082309 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:01:22.082773 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:22.082844 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:22.082978 25582 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:22.082976 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:22.084627 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.105105 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:22.162150 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:01:22.177212 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:22.188616 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:01:22.227840 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:22.256063 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:22.305051 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:22.313263 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:22.331208 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.336781 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:22.376454 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.385334 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.415949 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.453645 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.516396 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:22.576083 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:22.578621 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:22.584832 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:22.613914 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:22.617974 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.618497 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.631829 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:22.744309 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:22.757280 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:22.826483 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.838786 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:22.921558 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:22.928990 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:22.976353 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.012997 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.016244 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:23.040798 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.048324 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.051465 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:23.069183 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:01:23.082091 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.183084 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.205538 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.244218 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.244971 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.276170 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.320595 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.399273 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:23.404383 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.468272 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:23.476500 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.492188 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.502130 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:23.534945 7088 maintenance_manager.cc:419] P b417ed481cd9481ba8418c1658ec8f19: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033383
W20240503 02:01:23.541262 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:01:23.554752 7087 maintenance_manager.cc:643] P b417ed481cd9481ba8418c1658ec8f19: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.015s sys 0.003s Metrics: {"bytes_written":21490,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1038,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 02:01:23.555652 7088 maintenance_manager.cc:419] P b417ed481cd9481ba8418c1658ec8f19: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10880 bytes on disk
I20240503 02:01:23.556227 7087 maintenance_manager.cc:643] P b417ed481cd9481ba8418c1658ec8f19: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 02:01:23.578332 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:23.597445 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.659018 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.735021 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.753052 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.767597 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.830942 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:23.866104 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.890260 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:23.894424 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:23.900946 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:23.921713 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.979835 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:23.994918 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.004283 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:24.054033 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.072803 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.132314 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.135501 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.145714 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:24.245333 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.251359 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.317620 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.327364 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:24.364951 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.368965 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.389031 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:24.415760 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.469075 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:24.489243 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:01:24.516005 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:24.560969 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:24.601667 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.684971 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:24.746044 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:24.779372 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:24.780376 25179 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:24.780484 25179 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:24.780977 25179 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:24.781275 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:24.781419 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:24.781878 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.782011 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:24.782068 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:24.782191 25179 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:24.789124 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:24.806893 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.834961 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.872231 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:24.880345 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:24.885151 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.896653 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:24.931929 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:24.940747 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:25.020653 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.030051 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.062268 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.137048 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.184644 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:25.192628 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.201993 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.214870 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.219919 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.240520 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.248183 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:25.296663 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:25.331187 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.379738 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.550702 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.562464 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:25.620249 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.654453 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.658288 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.689847 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.715580 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.722481 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.728374 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:25.734285 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:25.758136 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.779323 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.798743 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.799650 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:25.810750 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:25.817960 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:25.843854 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:25.993942 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.028331 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:26.156234 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:26.208774 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.214773 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:26.216830 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:26.244328 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:26.259382 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.281735 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.286065 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:26.303846 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.339071 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.384430 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.405545 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 200 similar messages]
W20240503 02:01:26.418732 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:01:26.487947 25550 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:26.488077 25550 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:26.488555 25550 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:26.489053 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:01:26.489187 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:01:26.489866 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:26.489998 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:26.490051 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:26.490200 25550 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:26.508843 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:26.529358 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.562271 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.599759 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:26.609961 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:01:26.653679 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:26.656831 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.657872 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.724054 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.724138 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.766913 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.817142 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:26.825281 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.831035 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.842535 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:26.864245 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.933126 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:26.933360 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:26.940266 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:27.041590 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.134014 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:27.148624 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:01:27.162732 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:27.167088 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.172681 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.200407 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:01:27.293103 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.302105 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.338953 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.339527 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.392288 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:27.407662 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.410701 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:27.414877 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.449666 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.475958 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.479393 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.560691 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.561861 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:27.602820 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.648051 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.650200 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.665849 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:27.733214 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.778620 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.800626 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.863907 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:27.876132 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:27.951063 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:27.965154 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:27.974354 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.004992 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:28.012238 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.040278 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.053916 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:28.070537 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:01:28.084167 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:28.094686 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:28.190939 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.194208 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:28.250072 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.281592 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.294445 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:28.362303 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:28.401360 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:28.412639 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:28.462591 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:28.472167 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:28.491698 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:28.532374 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:28.555729 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.555971 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:28.575584 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:28.700875 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.715965 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:28.753633 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:28.773497 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:28.782400 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(18468345c7924d4f88366524b56e8ab3): perf score=0.033350
I20240503 02:01:28.807386 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(9796308eacd94fb5ad3cb524aa590cfa): perf score=0.033358
I20240503 02:01:28.816465 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(18468345c7924d4f88366524b56e8ab3) complete. Timing: real 0.034s user 0.032s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":169,"dirs.queue_time_us":200,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":981,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":291,"threads_started":3,"wal-append.queue_time_us":209}
I20240503 02:01:28.817755 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3): 4583 bytes on disk
I20240503 02:01:28.819365 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:01:28.844516 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:28.845965 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(9796308eacd94fb5ad3cb524aa590cfa) complete. Timing: real 0.038s user 0.032s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":5541,"dirs.queue_time_us":243,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1709,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":247,"threads_started":3,"wal-append.queue_time_us":146}
I20240503 02:01:28.847456 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(9796308eacd94fb5ad3cb524aa590cfa): 4838 bytes on disk
I20240503 02:01:28.848316 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(9796308eacd94fb5ad3cb524aa590cfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 02:01:28.849164 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(68791b3adccd4168b55a2da4423232d7): perf score=0.033366
I20240503 02:01:28.852644 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(18468345c7924d4f88366524b56e8ab3): perf score=0.033370
I20240503 02:01:28.870420 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(9796308eacd94fb5ad3cb524aa590cfa): perf score=0.033375
W20240503 02:01:28.881774 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:28.885010 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(68791b3adccd4168b55a2da4423232d7) complete. Timing: real 0.036s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":10131}
I20240503 02:01:28.886025 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7): 4923 bytes on disk
I20240503 02:01:28.886485 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 02:01:28.907593 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:28.922128 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(18468345c7924d4f88366524b56e8ab3) complete. Timing: real 0.069s user 0.027s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":222,"dirs.queue_time_us":7274,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":2617,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":310,"threads_started":3,"wal-append.queue_time_us":21904}
I20240503 02:01:28.923462 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3): 4583 bytes on disk
I20240503 02:01:28.924118 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 02:01:28.928772 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(9796308eacd94fb5ad3cb524aa590cfa) complete. Timing: real 0.058s user 0.028s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":11385,"dirs.queue_time_us":17496,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1783,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":225,"threads_started":3,"wal-append.queue_time_us":15524}
I20240503 02:01:28.930018 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(9796308eacd94fb5ad3cb524aa590cfa): 4838 bytes on disk
I20240503 02:01:28.930626 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(9796308eacd94fb5ad3cb524aa590cfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 02:01:28.931504 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(68791b3adccd4168b55a2da4423232d7): perf score=0.033390
W20240503 02:01:28.953826 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:28.989356 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:28.993935 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(68791b3adccd4168b55a2da4423232d7) complete. Timing: real 0.062s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5528,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1347,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":122,"threads_started":1,"wal-append.queue_time_us":28134}
I20240503 02:01:28.995131 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7): 4923 bytes on disk
I20240503 02:01:28.995609 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240503 02:01:29.044615 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.049934 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.097473 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:29.161885 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.181926 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:29.210705 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:29.239506 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.270103 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.278097 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.359452 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.362569 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:29.388749 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(81086ff737c84d9d9f04743d31c6151b): perf score=0.033343
W20240503 02:01:29.399183 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:29.402673 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(81086ff737c84d9d9f04743d31c6151b) complete. Timing: real 0.014s user 0.011s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1342,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":155}
I20240503 02:01:29.403574 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b): 4923 bytes on disk
I20240503 02:01:29.403959 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:01:29.404901 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850): perf score=0.033346
I20240503 02:01:29.420991 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1281,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":530}
I20240503 02:01:29.422068 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850): 4583 bytes on disk
I20240503 02:01:29.422523 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 02:01:29.438957 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.468618 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:29.497699 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(97ffb59d3ab64d03aca5d30a7c3be1e1): perf score=0.033374
W20240503 02:01:29.501916 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:29.510385 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(97ffb59d3ab64d03aca5d30a7c3be1e1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 02:01:29.511224 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(97ffb59d3ab64d03aca5d30a7c3be1e1): 4838 bytes on disk
I20240503 02:01:29.511597 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(97ffb59d3ab64d03aca5d30a7c3be1e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240503 02:01:29.520171 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.541453 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:29.549424 26478 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:29.549557 26478 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
W20240503 02:01:29.549963 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:29.550026 26478 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:29.550443 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:29.550585 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:29.551041 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.551342 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:29.551393 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:29.551494 26478 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:29.567255 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:29.572731 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(81086ff737c84d9d9f04743d31c6151b): perf score=0.033395
I20240503 02:01:29.584968 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(81086ff737c84d9d9f04743d31c6151b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1293,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":419,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":122}
I20240503 02:01:29.585755 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850): perf score=0.033398
I20240503 02:01:29.596139 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.010s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":1142,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":380,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":178}
I20240503 02:01:29.596908 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b): 4923 bytes on disk
I20240503 02:01:29.597288 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 02:01:29.598194 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850): 4583 bytes on disk
I20240503 02:01:29.598521 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 02:01:29.600085 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.610867 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:29.620332 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:01:29.662554 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.739988 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:29.824168 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:29.847029 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.880060 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.885221 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.931166 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.931746 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:29.942950 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:29.965899 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:29.966336 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:29.969228 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:29.981559 26726 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:29.981690 26726 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:29.982278 26726 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:29.982717 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:01:29.982901 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:01:29.983580 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:29.983671 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:29.983747 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:29.983883 26726 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:29.989236 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.022279 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:30.082616 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:30.128050 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.146777 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:30.175431 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(c95d3c29bfbd4899b27b770bf9d3f4a6): perf score=0.033360
W20240503 02:01:30.183521 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.184177 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:30.187170 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(c95d3c29bfbd4899b27b770bf9d3f4a6) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":208,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":184}
I20240503 02:01:30.187973 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(c95d3c29bfbd4899b27b770bf9d3f4a6): 4583 bytes on disk
I20240503 02:01:30.188333 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(c95d3c29bfbd4899b27b770bf9d3f4a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240503 02:01:30.209623 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:30.264348 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(6649e1feab5b40738003119a9122f07a): perf score=0.033387
I20240503 02:01:30.277992 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 02:01:30.278895 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(5f35ccb9c21b4c10b9187596d2f6929c): perf score=0.033389
W20240503 02:01:30.289832 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:30.290756 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(5f35ccb9c21b4c10b9187596d2f6929c) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":166}
I20240503 02:01:30.291539 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a): 4923 bytes on disk
I20240503 02:01:30.291893 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 02:01:30.292579 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(5f35ccb9c21b4c10b9187596d2f6929c): 4838 bytes on disk
I20240503 02:01:30.292910 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(5f35ccb9c21b4c10b9187596d2f6929c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240503 02:01:30.295408 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:30.368000 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.406641 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:30.418653 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:30.474925 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:30.490795 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:30.495132 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:30.520354 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.545521 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:30.557525 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:30.558121 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:30.560252 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:30.673063 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:01:30.680719 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(fb00fcf563b74243b7c955ddacf1d6cf): perf score=0.033343
W20240503 02:01:30.681833 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:30.690462 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(be1f4d2622c6452b869f88a301a5e9bc): perf score=0.033346
I20240503 02:01:30.693558 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(fb00fcf563b74243b7c955ddacf1d6cf) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":240,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1484,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":444,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":155}
I20240503 02:01:30.694375 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(fb00fcf563b74243b7c955ddacf1d6cf): 4923 bytes on disk
I20240503 02:01:30.694756 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(fb00fcf563b74243b7c955ddacf1d6cf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 02:01:30.695595 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(40260acad62242c3a72197d2dafae187): perf score=0.033346
I20240503 02:01:30.702947 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(be1f4d2622c6452b869f88a301a5e9bc) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":139}
I20240503 02:01:30.704043 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(be1f4d2622c6452b869f88a301a5e9bc): 4838 bytes on disk
I20240503 02:01:30.704433 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(be1f4d2622c6452b869f88a301a5e9bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 02:01:30.708189 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":357,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":146}
I20240503 02:01:30.708915 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187): 4583 bytes on disk
I20240503 02:01:30.709240 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 02:01:30.748965 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:30.782912 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:30.823135 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:30.853605 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(fb00fcf563b74243b7c955ddacf1d6cf): perf score=0.033394
I20240503 02:01:30.866463 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(fb00fcf563b74243b7c955ddacf1d6cf) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1379,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":439,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":142}
I20240503 02:01:30.867632 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(fb00fcf563b74243b7c955ddacf1d6cf): 4923 bytes on disk
I20240503 02:01:30.868046 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(fb00fcf563b74243b7c955ddacf1d6cf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 02:01:30.868695 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(40260acad62242c3a72197d2dafae187): perf score=0.033397
I20240503 02:01:30.879889 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.011s user 0.010s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1247,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":141}
I20240503 02:01:30.880780 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187): 4583 bytes on disk
I20240503 02:01:30.881412 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 02:01:30.884723 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.912204 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:30.958197 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.963871 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:30.972072 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:31.006062 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:31.037842 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:31.078701 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:31.136180 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:31.167594 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.177958 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:31.211278 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(8ebe541161c84bd6b11bcd3d13d3bd2e): perf score=0.033344
I20240503 02:01:31.224511 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(8ebe541161c84bd6b11bcd3d13d3bd2e) complete. Timing: real 0.013s user 0.006s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":195,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1395,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 02:01:31.225276 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e): 4838 bytes on disk
I20240503 02:01:31.225596 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 02:01:31.248834 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:31.297101 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(0d00b8fcb29f47298643903ec0cba606): perf score=0.033373
W20240503 02:01:31.297978 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:31.313704 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(0d00b8fcb29f47298643903ec0cba606) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3066,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":395}
I20240503 02:01:31.314718 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(a432cd5fe20941d6bc73dae386d85170): perf score=0.033376
I20240503 02:01:31.328441 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(a432cd5fe20941d6bc73dae386d85170) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1210,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":172}
I20240503 02:01:31.329581 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606): 4921 bytes on disk
I20240503 02:01:31.330116 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.330956 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(a432cd5fe20941d6bc73dae386d85170): 4581 bytes on disk
I20240503 02:01:31.331337 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(a432cd5fe20941d6bc73dae386d85170) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.383731 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(0d00b8fcb29f47298643903ec0cba606): perf score=0.033394
I20240503 02:01:31.395740 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(0d00b8fcb29f47298643903ec0cba606) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1385,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":143}
W20240503 02:01:31.396874 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:31.397035 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606): 4921 bytes on disk
I20240503 02:01:31.397431 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.398255 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(8ebe541161c84bd6b11bcd3d13d3bd2e): perf score=0.033396
W20240503 02:01:31.401317 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.409397 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 187 similar messages]
I20240503 02:01:31.411124 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(8ebe541161c84bd6b11bcd3d13d3bd2e) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":366}
I20240503 02:01:31.412024 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e): 4838 bytes on disk
I20240503 02:01:31.412441 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240503 02:01:31.495391 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:31.531456 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:31.541926 26478 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:31.542066 26478 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:31.542577 26478 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:31.542845 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:31.542995 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:31.543579 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:31.543833 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:31.543915 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:31.544181 26478 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:31.586441 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:31.591765 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.607327 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:31.616776 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:31.617071 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240503 02:01:31.650180 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.660581 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:31.724221 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.773561 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:31.786013 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.807374 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:31.833704 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(245cba4e5728491896e7995ca0d59b36): perf score=0.033358
W20240503 02:01:31.835458 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:31.846112 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":492,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":223,"threads_started":2,"wal-append.queue_time_us":230}
I20240503 02:01:31.846899 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36): 4838 bytes on disk
I20240503 02:01:31.847213 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240503 02:01:31.854750 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:31.874953 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:31.888300 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:31.914695 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(245cba4e5728491896e7995ca0d59b36): perf score=0.033381
I20240503 02:01:31.929512 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":128}
I20240503 02:01:31.930325 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36): 4838 bytes on disk
I20240503 02:01:31.930735 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.958868 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(f0539dded7554723b9e918e8cdfbab20): perf score=0.033394
I20240503 02:01:31.976639 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(f0539dded7554723b9e918e8cdfbab20) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":148}
I20240503 02:01:31.977475 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20): 4921 bytes on disk
I20240503 02:01:31.977849 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.978003 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(f0539dded7554723b9e918e8cdfbab20): perf score=0.033401
I20240503 02:01:31.989879 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(f0539dded7554723b9e918e8cdfbab20) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1413,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":388,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":142}
I20240503 02:01:31.990701 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20): 4921 bytes on disk
I20240503 02:01:31.991055 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 02:01:31.991684 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(3b15d48b2fa04269b98c6f801f54df4f): perf score=0.033404
I20240503 02:01:32.001861 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(3b15d48b2fa04269b98c6f801f54df4f) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":337,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":157}
I20240503 02:01:32.002555 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(3b15d48b2fa04269b98c6f801f54df4f): 4583 bytes on disk
I20240503 02:01:32.002878 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(3b15d48b2fa04269b98c6f801f54df4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240503 02:01:32.003434 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(245cba4e5728491896e7995ca0d59b36): perf score=0.033405
W20240503 02:01:32.016824 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:32.017411 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.014s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1396,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":149}
I20240503 02:01:32.018218 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36): 4838 bytes on disk
I20240503 02:01:32.018613 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240503 02:01:32.056895 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.125850 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.138545 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.144500 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:32.155651 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:01:32.156080 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:32.186033 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:32.209791 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.226065 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:01:32.226738 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.281232 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.295717 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.333557 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:32.349363 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(ec179e5e683d4ba099c459331ea5e578): perf score=0.033343
W20240503 02:01:32.354041 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:32.364789 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(ec179e5e683d4ba099c459331ea5e578) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1523,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":159}
I20240503 02:01:32.365628 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(8ad24b47690b4263bdf4a9157db7c17f): perf score=0.033347
I20240503 02:01:32.377377 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(8ad24b47690b4263bdf4a9157db7c17f) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":379,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1920,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":174}
I20240503 02:01:32.378170 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f): 4923 bytes on disk
I20240503 02:01:32.378508 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240503 02:01:32.379151 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578): 4838 bytes on disk
I20240503 02:01:32.379528 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 02:01:32.418998 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:32.421114 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.428009 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:32.432888 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(8ad24b47690b4263bdf4a9157db7c17f): perf score=0.033363
I20240503 02:01:32.444618 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(8ad24b47690b4263bdf4a9157db7c17f) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1431,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":407,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":149}
I20240503 02:01:32.445453 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486): perf score=0.033365
I20240503 02:01:32.456718 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":395,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":289}
I20240503 02:01:32.457587 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f): 4923 bytes on disk
I20240503 02:01:32.457983 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 02:01:32.458700 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486): 4583 bytes on disk
I20240503 02:01:32.459024 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 02:01:32.466461 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:32.480068 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(ec179e5e683d4ba099c459331ea5e578): perf score=0.033380
I20240503 02:01:32.502424 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(ec179e5e683d4ba099c459331ea5e578) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":7228,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":195}
I20240503 02:01:32.503408 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486): perf score=0.033385
W20240503 02:01:32.512255 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:32.521338 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486): perf score=0.033392
I20240503 02:01:32.523535 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.020s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1285,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":154}
I20240503 02:01:32.524811 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578): 4838 bytes on disk
I20240503 02:01:32.525168 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 02:01:32.526568 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486): 4583 bytes on disk
I20240503 02:01:32.526947 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 02:01:32.529239 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:32.536705 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1285,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":118}
I20240503 02:01:32.537413 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486): 4583 bytes on disk
I20240503 02:01:32.537744 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240503 02:01:32.543490 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.610141 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:32.632565 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:32.660780 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.694597 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.699003 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:32.706810 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:32.722211 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:32.763927 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:32.813139 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:01:32.881702 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(7d332a7c255b44589d54b1f583a2f9c4): perf score=0.033353
I20240503 02:01:32.893440 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(7d332a7c255b44589d54b1f583a2f9c4) complete. Timing: real 0.012s user 0.007s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":476,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":134}
W20240503 02:01:32.894373 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:32.894424 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(7d332a7c255b44589d54b1f583a2f9c4): 4923 bytes on disk
I20240503 02:01:32.894866 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(7d332a7c255b44589d54b1f583a2f9c4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 02:01:32.961186 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(c18129344d124a5f8f688ec35ef69a55): perf score=0.033376
W20240503 02:01:32.970679 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.977540 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:32.978192 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:32.979096 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(c18129344d124a5f8f688ec35ef69a55) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3991,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1525,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":1597}
I20240503 02:01:32.980123 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55): 4583 bytes on disk
I20240503 02:01:32.980639 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.007877 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:33.008118 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:33.028376 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:33.029412 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(cc68f31e6ef8411e96d46b2419bf9676): perf score=0.033395
I20240503 02:01:33.039852 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(c18129344d124a5f8f688ec35ef69a55): perf score=0.033396
I20240503 02:01:33.045913 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(cc68f31e6ef8411e96d46b2419bf9676) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1355,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":506,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 02:01:33.047181 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(cc68f31e6ef8411e96d46b2419bf9676): 4836 bytes on disk
I20240503 02:01:33.047698 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(cc68f31e6ef8411e96d46b2419bf9676) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.048456 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:01:33.055126 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(c18129344d124a5f8f688ec35ef69a55) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1401,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":149}
I20240503 02:01:33.056109 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55): 4583 bytes on disk
I20240503 02:01:33.056555 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.080029 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:33.087116 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.097481 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:01:33.116153 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:33.117329 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:33.182113 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.264389 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:33.280076 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.293429 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:33.299085 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(71f12a25a1db4650a6c37b5000a668c7): perf score=0.033335
W20240503 02:01:33.307087 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:33.307992 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(527af716906b42729952db28e790bafd): perf score=0.033338
I20240503 02:01:33.311013 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(71f12a25a1db4650a6c37b5000a668c7) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":149}
I20240503 02:01:33.311890 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(527af716906b42729952db28e790bafd): perf score=0.033337
W20240503 02:01:33.316984 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:01:33.320822 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(527af716906b42729952db28e790bafd) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1221,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":152}
I20240503 02:01:33.321699 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd): 4839 bytes on disk
I20240503 02:01:33.322048 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.324553 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(527af716906b42729952db28e790bafd) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1000,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":105,"threads_started":1,"wal-append.queue_time_us":184}
I20240503 02:01:33.325342 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd): 4839 bytes on disk
I20240503 02:01:33.325734 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.326376 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7): 4583 bytes on disk
I20240503 02:01:33.326665 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.396919 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(a0a526f7fced4a04b4e0327f4656cf69): perf score=0.033362
I20240503 02:01:33.409107 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(a0a526f7fced4a04b4e0327f4656cf69) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":146}
I20240503 02:01:33.409868 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69): 4921 bytes on disk
I20240503 02:01:33.410233 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.410845 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(71f12a25a1db4650a6c37b5000a668c7): perf score=0.033365
I20240503 02:01:33.421665 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(71f12a25a1db4650a6c37b5000a668c7) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1301,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":407,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":151}
I20240503 02:01:33.422452 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7): 4583 bytes on disk
I20240503 02:01:33.422792 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.459197 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:33.473178 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:33.482933 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(a0a526f7fced4a04b4e0327f4656cf69): perf score=0.033389
W20240503 02:01:33.489976 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:33.495976 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(a0a526f7fced4a04b4e0327f4656cf69) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":162}
I20240503 02:01:33.496778 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69): 4921 bytes on disk
I20240503 02:01:33.497118 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.499873 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:33.553625 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.605440 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:33.716095 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:33.720402 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.721019 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:33.755784 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:33.804524 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.819805 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:01:33.824095 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(27c539b143674bdd8e3e5aaa4b78c1e9): perf score=0.033342
I20240503 02:01:33.828517 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(a0b1a4b2ede6493f9e4e094649b4b57d): perf score=0.033345
I20240503 02:01:33.835459 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(27c539b143674bdd8e3e5aaa4b78c1e9) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":401,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":164}
I20240503 02:01:33.836509 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(66317a254eb644359f65610368d84d76): perf score=0.033345
I20240503 02:01:33.839988 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(a0b1a4b2ede6493f9e4e094649b4b57d) complete. Timing: real 0.011s user 0.010s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":141}
I20240503 02:01:33.840785 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d): 4923 bytes on disk
I20240503 02:01:33.841122 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.841734 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(27c539b143674bdd8e3e5aaa4b78c1e9): perf score=0.033347
I20240503 02:01:33.848644 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(66317a254eb644359f65610368d84d76) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1184,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":150}
I20240503 02:01:33.849582 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76): 4838 bytes on disk
I20240503 02:01:33.849952 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.850855 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9): 4581 bytes on disk
I20240503 02:01:33.851179 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 02:01:33.853760 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(27c539b143674bdd8e3e5aaa4b78c1e9) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":1128,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":162}
I20240503 02:01:33.854598 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9): 4581 bytes on disk
I20240503 02:01:33.854925 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.890938 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:33.924957 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(a0b1a4b2ede6493f9e4e094649b4b57d): perf score=0.033374
W20240503 02:01:33.931967 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:33.940586 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(a0b1a4b2ede6493f9e4e094649b4b57d) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2076,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":173}
I20240503 02:01:33.941581 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d): 4923 bytes on disk
I20240503 02:01:33.941964 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 02:01:33.947247 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:33.953687 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:01:33.999109 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(66317a254eb644359f65610368d84d76): perf score=0.033394
W20240503 02:01:34.004735 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:34.010792 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(66317a254eb644359f65610368d84d76) complete. Timing: real 0.011s user 0.009s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1186,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":388,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 02:01:34.011580 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76): 4838 bytes on disk
I20240503 02:01:34.011902 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 02:01:34.053886 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.109792 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.111985 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:34.119529 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.162912 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:34.181903 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.261947 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:34.283713 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:34.330857 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:34.348551 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.365672 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:34.369210 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:34.401286 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:34.415176 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:01:34.485113 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.495158 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:34.514103 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:34.531948 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.542008 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:34.552263 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:34.643584 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:34.696209 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:34.702189 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.795291 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.805285 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:34.885344 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:34.900832 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.934195 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:34.943421 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:34.970862 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:34.996369 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.044807 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:35.065706 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.069937 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:35.077428 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.081156 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:35.082211 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.168375 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.175252 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.187904 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.300014 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.310304 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:35.350116 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.366706 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.445962 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.448719 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.459493 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.461529 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.484820 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.509727 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.585832 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.630815 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:35.636574 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.655911 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:35.668591 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:35.677217 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.691574 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:35.768865 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.830680 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.843392 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.854099 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:35.870975 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:35.881443 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:35.935443 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.001564 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.020323 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:36.102396 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:36.148458 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:36.174012 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:36.177289 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:36.186412 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.193581 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.253656 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.344429 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.398679 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:36.413656 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 187 similar messages]
W20240503 02:01:36.422888 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.450536 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:36.480942 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.499444 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:36.522068 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.568676 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:36.635103 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:01:36.635808 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:36.665902 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.746423 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:36.769024 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.803583 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.824065 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:01:36.841945 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:36.934007 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.936237 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:36.961920 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:36.974598 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:36.977622 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:36.980206 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:37.036314 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.045102 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.065603 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:37.094398 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:37.162166 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.256356 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.260807 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:37.260807 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.265182 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:01:37.266595 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.315400 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.326823 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.345367 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.392387 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.454684 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.477428 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:37.550293 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:37.578100 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:37.614579 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:37.617106 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.656456 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:37.677140 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.690344 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.730978 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.741571 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.748135 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.773741 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:37.776970 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:37.783993 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:37.809957 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:37.828644 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.918615 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.922600 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.950577 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.961768 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:37.994371 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:38.011353 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:38.038205 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.038295 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:38.060178 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.104873 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:01:38.161410 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.173960 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.185842 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:01:38.203902 27758 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:38.204092 27758 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:38.204648 27758 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:38.204946 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:38.205096 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:38.205576 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.205698 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:38.205767 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:38.205889 27758 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:38.252588 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.262480 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.278640 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:38.318984 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.353121 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.385849 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:38.467916 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:38.504499 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:38.518798 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.571396 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.582394 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.642549 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:38.663790 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:38.671202 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.692343 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:38.702698 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:38.751945 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:38.825563 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:38.856657 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:01:38.867064 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:38.867201 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:38.867897 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:38.868144 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:01:38.868284 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:01:38.868816 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:38.868932 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:38.868981 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:38.869117 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:38.895596 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:38.975124 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:38.985421 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:39.002005 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:39.023559 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.027927 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:39.031391 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.106834 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.131861 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.147749 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:39.226411 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:39.259415 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:39.266525 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.285126 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:01:39.341176 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.468920 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:39.495947 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.516924 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:39.534797 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:39.548651 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.568377 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.573055 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.583851 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.634790 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.687119 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.699451 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.720048 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:39.740787 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:39.849313 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:39.903229 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:39.953325 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.015591 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.017659 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.020835 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:40.043857 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:40.077934 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.110726 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.187428 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.216185 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:40.232283 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.255613 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.280007 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.300228 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.328825 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.357650 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:40.361654 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.362588 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.363737 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.398952 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.414705 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:40.425647 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.435225 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.480391 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.484838 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.515136 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.520431 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.544919 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.547035 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.581872 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:40.602558 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:40.661778 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.661784 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.667692 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:40.683153 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.753089 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:40.933679 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:40.933763 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:40.972508 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:40.980444 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:41.050415 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.057873 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.107790 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.127750 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.157243 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:41.226078 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:41.274322 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:41.284363 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(c95d3c29bfbd4899b27b770bf9d3f4a6): perf score=0.033352
I20240503 02:01:41.295584 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(c95d3c29bfbd4899b27b770bf9d3f4a6) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":212,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":384,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":165}
I20240503 02:01:41.296543 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(6649e1feab5b40738003119a9122f07a): perf score=0.033347
I20240503 02:01:41.307837 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":371,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":148}
I20240503 02:01:41.308660 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a): 4923 bytes on disk
I20240503 02:01:41.308979 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 02:01:41.309576 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(c95d3c29bfbd4899b27b770bf9d3f4a6): 4583 bytes on disk
I20240503 02:01:41.309847 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(c95d3c29bfbd4899b27b770bf9d3f4a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
W20240503 02:01:41.314249 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.335851 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:01:41.373682 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(6649e1feab5b40738003119a9122f07a): perf score=0.033375
I20240503 02:01:41.376137 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling FlushMRSOp(a432cd5fe20941d6bc73dae386d85170): perf score=0.033370
I20240503 02:01:41.391464 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":252,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":135}
I20240503 02:01:41.392318 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a): 4923 bytes on disk
I20240503 02:01:41.392575 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: FlushMRSOp(a432cd5fe20941d6bc73dae386d85170) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":571776,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 02:01:41.392767 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(6649e1feab5b40738003119a9122f07a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 02:01:41.393404 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850): perf score=0.033366
I20240503 02:01:41.393406 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(a432cd5fe20941d6bc73dae386d85170): 4581 bytes on disk
I20240503 02:01:41.393780 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(a432cd5fe20941d6bc73dae386d85170) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 02:01:41.405377 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: FlushMRSOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1249,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":154}
I20240503 02:01:41.406388 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850): 4583 bytes on disk
I20240503 02:01:41.406773 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(b4628afbfeff4f8bbaba5da1e89e4850) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240503 02:01:41.413174 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:41.429569 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:41.430033 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 193 similar messages]
W20240503 02:01:41.430730 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.499881 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.522718 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:41.540755 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:41.549441 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:01:41.555748 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.584237 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.586710 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.587890 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:41.601783 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:41.632530 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.639600 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:01:41.643289 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:41.808367 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:41.809332 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:41.811904 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling FlushMRSOp(5f35ccb9c21b4c10b9187596d2f6929c): perf score=0.033385
I20240503 02:01:41.824302 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: FlushMRSOp(5f35ccb9c21b4c10b9187596d2f6929c) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":142}
I20240503 02:01:41.825183 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(5f35ccb9c21b4c10b9187596d2f6929c): 4838 bytes on disk
I20240503 02:01:41.825610 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(5f35ccb9c21b4c10b9187596d2f6929c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 02:01:41.901300 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.939481 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:41.954783 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:41.959836 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:41.967311 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.055464 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.094137 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.115357 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:42.130014 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:42.172791 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:01:42.216076 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling FlushMRSOp(40260acad62242c3a72197d2dafae187): perf score=0.033398
I20240503 02:01:42.228034 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: FlushMRSOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":145}
I20240503 02:01:42.228955 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187): 4583 bytes on disk
I20240503 02:01:42.229344 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 02:01:42.252750 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:42.326211 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.328142 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:01:42.363936 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.410867 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.420436 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.428962 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.475745 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.499639 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.531788 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:42.544934 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:42.588626 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.598110 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:42.605346 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:42.644143 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:42.679940 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:42.698377 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:42.740474 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.767066 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:42.782117 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.893003 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.924939 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.939474 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.945719 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.979511 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:42.980994 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:42.995630 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.021806 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.035990 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:43.036932 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.055922 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:43.093199 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:43.100606 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:43.114945 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:43.119724 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.144305 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:43.161772 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:01:43.171479 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:43.182773 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.225437 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.304167 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.304459 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.373788 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.439159 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.454649 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.479168 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.543077 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:43.564978 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:43.617890 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:43.621448 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:43.740994 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:43.746084 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.793987 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:43.864892 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:43.872344 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.904958 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:43.944806 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:43.977327 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:43.985428 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:44.001683 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.034492 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:44.060266 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.106920 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:44.123293 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.140957 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:44.186758 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:44.189465 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.231009 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:01:44.273350 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:44.330161 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.353909 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:44.377715 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.438292 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.446883 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:44.578759 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.579488 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.594192 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.595821 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:44.641867 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:44.672761 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:44.680014 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.682675 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.710526 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.779006 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.831496 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:44.873919 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.901144 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:44.908028 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:44.956380 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:44.972929 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:45.039014 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.059314 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.061756 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.113014 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:45.191263 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:45.198798 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.218398 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:45.313129 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.324872 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.325354 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.338344 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.353085 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:45.389047 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.391986 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:45.417891 28122 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:45.418018 28122 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:45.419178 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:01:45.419337 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
I20240503 02:01:45.418571 28122 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
W20240503 02:01:45.420126 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:45.420414 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:45.420475 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:45.420754 28122 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:45.423398 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.458776 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.472610 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:45.483479 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.492614 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.554648 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.554915 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:45.579352 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.586007 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:45.618460 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:45.764501 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:45.796854 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.798106 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.808221 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:45.836997 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:45.839221 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.860811 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.865568 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:45.886672 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:45.973493 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:46.014910 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:46.019879 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.163861 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.197382 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:46.216058 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.242118 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:46.273252 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.283363 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:46.305125 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:46.355155 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:46.394759 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.400257 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.412568 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:46.426425 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.437285 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:46.446281 7567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:01:46.538987 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.639217 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:46.644358 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 02:01:46.655715 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:46.705821 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:46.747344 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.754314 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:46.764330 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.785600 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:01:46.826432 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.906446 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:46.908084 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:46.954057 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:46.969628 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:46.994570 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:47.072883 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:01:47.073678 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:47.079207 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.095752 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:47.132759 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.179150 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.190531 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:47.244038 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.319394 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.351663 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:01:47.352346 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.377837 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:47.405761 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:47.407950 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.417403 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.424880 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.442276 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.479020 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:47.520540 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:47.610231 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:47.611135 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:47.615996 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:47.641674 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:47.691632 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.707134 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.803110 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.831506 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.836520 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.903074 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.913269 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.914327 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.914383 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:47.931852 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:47.936668 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:47.985904 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:48.004359 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:48.040352 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.065814 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.107064 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:48.121809 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:48.139123 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.139914 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:48.188108 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:48.193418 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:48.255462 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.264855 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.270567 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.295786 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.364391 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:48.412338 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.472738 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.477543 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:48.509171 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.518085 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:48.556594 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.578253 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:48.704716 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:48.716398 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.750380 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.753441 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:48.764555 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:48.835067 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.883206 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.936378 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:01:48.946183 29185 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:48.946313 29185 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:48.946882 29185 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:48.947382 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:01:48.947576 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:01:48.948174 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:48.948253 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:48.948400 29185 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:48.948563 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:48.953037 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:48.976114 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:48.986793 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:48.990718 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:49.010862 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.020753 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:49.035039 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.044349 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:49.159670 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:01:49.210294 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:49.293360 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.306915 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:49.314445 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:01:49.318825 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:49.318954 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:49.319401 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:49.319787 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:01:49.319965 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:01:49.320678 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:49.320768 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:49.320853 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:49.320986 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:49.348783 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.421149 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.460076 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:49.496683 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:49.523164 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:49.557817 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.589401 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.602708 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:49.603307 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:49.615942 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.686527 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:49.739485 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.800004 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:49.854616 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.901161 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:49.966904 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:49.973549 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.000114 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.046682 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:50.066318 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.069021 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.106215 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.116540 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.136344 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.148499 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:50.157989 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:50.161489 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.184051 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:50.198007 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.269210 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.273819 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.276305 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.301208 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.370123 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.418814 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.422416 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.485934 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.494544 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.511289 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.557220 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:50.572881 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:50.581086 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.592250 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:50.647282 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.678615 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:50.685833 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:50.700631 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:50.719615 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.738071 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.855121 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.886662 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:50.984740 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:51.037021 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.046975 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.062788 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:51.082207 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.105576 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.106020 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:51.123754 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.178875 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:51.214470 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.292641 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.391733 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:51.428056 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.429118 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:51.433228 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:51.436349 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.447794 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:01:51.515174 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:51.518713 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:51.554319 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:51.632145 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:51.636024 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.638659 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:01:51.650350 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:01:51.652078 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:51.679657 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.681483 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:51.722618 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:51.739531 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:51.854290 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:51.879998 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:51.923058 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:51.978251 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:52.008379 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:52.011716 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.012199 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.040792 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.055430 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.120556 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.189345 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:52.291850 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.316516 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.358747 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:52.387231 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.402928 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.423976 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:01:52.448768 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.462996 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.483186 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.502090 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.502908 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:52.515842 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.548524 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.567016 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.609771 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.628628 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:52.666334 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.671311 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.675482 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:52.691790 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.726796 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:52.746248 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:52.749727 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.875165 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.894682 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.920398 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:52.967856 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:52.974087 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:53.016618 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.084093 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:53.096799 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.104633 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:53.148905 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:53.159502 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.166991 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:53.175515 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.183740 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.185586 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:53.213040 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:01:53.254863 29862 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:01:53.255007 29862 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:01:53.255641 29862 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:01:53.256069 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:01:53.256239 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:01:53.256743 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.257030 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:01:53.257109 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:01:53.257249 29862 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:53.262378 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.268230 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:53.285246 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.443245 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:53.487578 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.504228 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:53.553014 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:01:53.588629 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.613348 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:53.619356 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:53.631888 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:53.646735 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:53.668443 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.743729 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.745599 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.871667 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:53.892946 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:53.923897 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:53.982553 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.018189 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.022765 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:54.027074 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:54.047618 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:54.089790 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:54.105165 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:54.109136 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.118656 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.134662 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:01:54.138146 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:54.176791 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:54.197008 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:54.293896 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.351465 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.371201 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.408483 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.463480 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:54.524920 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.583230 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:54.600320 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.656280 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:54.707865 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.725636 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:54.796793 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:54.873901 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:54.900045 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.907943 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:54.922039 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.985003 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:54.998226 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.024473 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:55.025012 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:55.027284 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:55.075206 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.080924 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:55.098176 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.101809 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.126993 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.150063 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.196260 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.220119 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:55.234438 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.244107 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:55.340734 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.371234 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.373764 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.442965 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:55.449918 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.480778 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.482170 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.537786 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.566475 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.575719 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:55.594208 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:55.602699 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:55.614012 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:55.633813 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:55.687074 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.694706 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.715507 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:55.752488 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:55.800062 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.837163 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:55.855392 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.024752 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:56.123893 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.131279 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:56.158010 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:56.160280 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:56.162104 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:56.173513 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.213881 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.236259 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:56.277225 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.289521 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.423108 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:56.441375 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:56.458079 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:01:56.565649 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:56.568351 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:56.570853 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.615720 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:56.624583 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:56.651644 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:56.661044 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.665254 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.665938 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:01:56.695628 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:56.696868 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:01:56.706283 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.766973 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:56.804255 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.921170 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:56.933287 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.945789 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:56.947475 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.952826 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.972585 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:56.988180 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.037642 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.138187 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.159626 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.193534 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.217272 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:57.260015 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.296259 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:57.309043 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.376713 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.383708 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:01:57.398403 29784 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:57.398540 29784 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:01:57.398990 29784 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:01:57.399806 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:01:57.399953 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:01:57.400043 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.400226 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:01:57.400296 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:01:57.400534 29784 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:57.419245 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:57.422204 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.508046 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 02:01:57.508733 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.508771 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:57.512233 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.516518 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.531437 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.543033 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.546242 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.613723 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.628542 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:57.630416 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.655472 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.672013 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.742738 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.781816 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:57.791569 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:57.899739 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:57.936713 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.944943 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.947500 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.950424 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:57.966924 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:57.996101 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:58.015782 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.030516 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.060060 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.116570 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:58.158121 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.195623 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:58.257329 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.265072 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.272225 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.287196 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.325037 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:58.392961 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:58.470227 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.477337 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:58.492494 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:58.517573 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:58.602351 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:01:58.602983 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:58.655690 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:58.669602 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:58.733418 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:58.782833 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.817320 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:58.878710 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:58.890502 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:58.923789 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:58.940618 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:58.988001 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.036089 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.081020 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:59.143747 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.145852 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:59.146438 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:59.178843 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:59.239176 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.309841 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.387089 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.396317 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:01:59.405692 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.449210 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.495131 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:01:59.510056 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:59.571270 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.572405 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:59.575527 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.578651 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.651717 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.700814 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:59.709915 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:59.744421 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.767690 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:01:59.771663 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:59.850912 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.851007 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.888041 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.928751 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.933564 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:01:59.935336 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:01:59.949559 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.960218 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:01:59.967653 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:01:59.969594 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:00.107595 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.141924 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.166811 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.196501 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:00.205540 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:00.205951 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.324448 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:00.331627 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.334463 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:00.369622 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.419306 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.458850 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:00.459868 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.509244 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:00.520983 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.526356 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:00.535292 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.543766 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.608671 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.650063 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:00.650149 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.654829 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.672222 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.754292 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.755424 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:00.788589 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:00.845413 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:00.864910 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:00.891602 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:00.989251 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:00.991328 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.033782 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:01.070426 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:01.076690 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:01.166764 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.191112 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.234411 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:01.313469 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:01.372604 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:01.411448 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:01.416358 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:01.476900 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:02:01.477504 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:01.513782 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:01.543736 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:02:01.546947 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:01.560289 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.636166 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:01.642176 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:01.648759 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:01.692745 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:02:01.709650 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.778291 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:01.858498 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:01.882856 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:01.906047 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:01.967715 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.028463 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.099488 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.125696 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.162720 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:02.169487 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:02.187784 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.199285 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.245283 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.298460 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.304241 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:02:02.313668 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:02.313791 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:02.314317 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:02.314973 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:02.315106 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:02.315424 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.316040 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:02:02.316139 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:02.316301 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:02.386353 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:02.389925 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.397086 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.399299 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.406267 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:02.419530 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.421415 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.422920 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.427284 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.524044 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:02:02.564966 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.618124 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.624316 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.649798 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.654054 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.659191 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:02.662057 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:02.702248 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.718427 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.742401 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:02.770217 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.845095 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.916756 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:02.917304 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:02.967356 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:02.981372 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.015480 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:03.035636 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.041070 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.072706 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.075259 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.079078 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:03.129891 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.135216 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.165232 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:03.180480 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.183506 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:03.228420 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:03.288409 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:03.325735 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.355386 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:03.406935 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:03.445492 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:03.477591 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.538359 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:03.540390 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.618455 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:02:03.619004 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:03.620754 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:03.691428 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:03.742417 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:03.753163 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:03.757076 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:03.824424 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:02:03.919760 31256 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:02:03.919899 31256 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:03.920270 31256 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:03.920624 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:02:03.920807 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:02:03.921255 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:03.921392 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:03.921469 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:02:03.921592 31256 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:03.992377 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.031383 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:04.050540 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.054850 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:04.082857 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.094094 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.118973 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:04.139137 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:04.144701 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.193428 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:02:04.195973 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:04.217839 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:04.258158 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.546617 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:04.567507 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:04.606469 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.616801 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.648039 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:04.676239 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.678751 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:04.707695 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.720430 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.729915 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:04.734714 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.748863 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:04.763584 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:04.778399 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.810693 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:04.945495 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.953882 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.954878 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.969465 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:04.989427 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.010046 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.011639 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.023098 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:05.065485 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.181819 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.190016 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:05.191485 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.200198 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:05.238344 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.238375 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.259805 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.285140 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:05.349983 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.405205 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.452015 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.481396 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.494058 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.518353 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.534292 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.550179 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:05.570631 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:05.594389 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:05.616946 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.621054 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.622208 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.646263 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:05.659001 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.675105 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:05.691205 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:05.707216 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.720786 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.735281 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:05.736773 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.830753 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.952385 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:05.965699 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:06.020399 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:06.049049 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:06.070765 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.156508 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.165488 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.184103 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:06.247331 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:06.306138 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:06.409494 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:06.449244 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:06.456079 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:06.463522 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:06.499944 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:02:06.522056 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:06.558640 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:06.582624 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:06.608991 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.611459 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.665877 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:06.712163 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:02:06.728356 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:06.730754 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:06.784821 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:06.853623 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:06.940384 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:06.951418 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:02:06.955472 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:06.967963 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.013502 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.098755 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:07.101384 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:07.114781 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.144505 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:07.153623 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:07.229007 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.236459 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.239852 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.250366 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.253706 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.358033 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.377830 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:07.443197 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.454975 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:07.471706 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.483909 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:07.484308 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:07.527313 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.536235 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:02:07.597816 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.644183 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.662732 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.696193 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.719488 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:07.744760 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:07.747125 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.777155 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:07.801331 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.838068 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:07.840538 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:07.906909 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:07.959444 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:08.004505 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:08.013911 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.040699 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.064888 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:08.065096 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:08.066447 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.114310 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.120426 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:08.124653 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.180004 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:08.196694 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:08.210783 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:08.225560 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.248195 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.252518 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:02:08.270335 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:08.270467 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:08.270920 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:08.271315 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:08.271471 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:08.272244 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.272327 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:02:08.272410 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:08.272536 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:08.301373 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.328188 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:08.378077 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.418277 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.437513 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.442153 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.566058 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:08.591076 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:08.659987 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.662212 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:02:08.684537 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:08.731601 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:08.792697 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:08.834743 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.868922 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:08.882647 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:08.950268 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:09.034631 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.052253 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:09.071343 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:09.122762 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.132259 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:09.161166 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.168341 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:09.182338 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.195755 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:09.273506 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:09.280114 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:09.330475 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:02:09.378077 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:09.390991 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.459908 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:09.488941 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:09.498171 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:09.500260 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.622385 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.622740 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:09.634380 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.648008 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.742065 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.781204 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:09.788277 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.798458 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.844936 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:09.852970 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:09.874408 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:09.888025 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.007022 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:10.010288 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.044374 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.121115 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.127528 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.178459 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.193181 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.199738 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:10.206416 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.256040 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.333055 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.333830 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.364058 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:10.374176 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.375087 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.404551 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.417779 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:10.466058 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.477550 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.489166 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.571177 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:10.573329 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.584997 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.653169 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.670480 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.682777 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.694787 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:10.723923 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:10.736441 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.748180 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.753017 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:10.763024 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:10.815194 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:10.906543 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.945427 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:10.972662 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:10.977763 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.067471 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:11.079383 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.084331 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.186781 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:11.222023 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.272538 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.279119 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:11.349032 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.391151 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:11.428938 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:11.448724 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:11.503216 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240503 02:02:11.537904 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.569957 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:11.571462 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.600910 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:02:11.666728 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:11.713151 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.757776 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:02:11.758571 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:11.760226 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:11.863471 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.866475 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.874491 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:11.902463 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:11.914762 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:11.953912 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:11.976807 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:12.008592 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.099619 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.112952 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.118999 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.156172 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:12.159197 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.164296 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.180603 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.264524 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:12.285130 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:12.297386 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.313012 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.364457 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.374922 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.384976 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.543596 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.549445 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:02:12.586465 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:12.588399 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.615309 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:12.623893 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.636265 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.648376 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:12.683699 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:12.689138 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.778671 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.786125 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.801790 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.824435 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.834827 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.923231 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.925165 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:12.925194 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.969378 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:12.984148 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:12.990410 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.003381 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:13.105919 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.110343 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.120060 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.125335 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.141595 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:13.154191 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.162657 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.164642 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:13.184051 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:13.184106 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:13.199630 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.208861 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:13.357398 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:13.403893 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.404000 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:13.438405 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:13.475179 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:13.589998 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.589991 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.595144 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:13.640139 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:13.650400 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:13.652015 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:13.760748 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.797391 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.807420 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:13.872550 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:13.910336 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:13.937047 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.018362 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:14.094094 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:02:14.094754 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:14.126657 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:14.182163 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:02:14.208223 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:14.220538 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.331650 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:14.343631 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.357834 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.422788 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:14.478224 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.492425 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.519060 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:14.519999 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:14.542968 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.593801 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.624493 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.626847 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.692348 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:14.702389 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:14.712862 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:14.751579 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.842592 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.901410 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.934286 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:14.937695 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:14.940836 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:14.971956 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:14.992673 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.054391 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.061635 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.066473 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.095135 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:15.142544 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.169714 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.183454 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.211922 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:15.276232 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.301151 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.302868 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.318259 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.321327 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.358050 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:15.385143 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:15.385324 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:15.385826 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:15.386332 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:15.386561 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:15.387084 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.387264 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:02:15.387331 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:15.387472 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:15.405982 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.420554 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.433841 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.447712 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.449962 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:15.470284 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.480623 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.481173 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.523705 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:15.538661 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:15.559896 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.616935 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.618614 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:15.653882 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.676972 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:15.721653 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.734934 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.791733 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:15.795821 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:15.874732 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.882598 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:15.892649 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:15.983184 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:16.013397 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:16.013437 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:16.040781 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:16.176240 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:16.202775 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:16.237433 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:16.294670 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:16.331123 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:16.397744 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:16.405297 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:16.410908 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:16.433156 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:16.446508 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:16.465725 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:16.503407 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:02:16.504076 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:16.527840 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:16.563387 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:16.587703 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:02:16.588752 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:16.759696 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:02:16.860056 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:16.875738 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:16.930763 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:16.949875 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:16.957904 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.008075 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.014078 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:17.016268 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.029901 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.071509 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.093175 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.109674 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:17.116148 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.164547 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.258098 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.258468 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.321363 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:17.355150 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.362720 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.387203 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.389050 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.394892 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:17.424422 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.436899 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:17.520507 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.568197 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.577746 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:17.614225 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.628393 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.677248 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.712096 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:02:17.712745 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.713672 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:17.763674 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.783742 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.792107 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:17.813622 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:17.814227 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:17.862679 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:17.879846 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.882174 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.917052 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:17.917050 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.919658 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:17.980494 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.001681 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.063500 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.102896 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.122206 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.137373 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.177443 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:18.193418 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.219731 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.287036 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.296548 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:18.309728 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:18.321519 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.329748 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.345950 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.374312 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.432271 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.434590 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.483176 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:18.671653 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:18.672423 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:18.677597 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.685848 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:18.699069 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:18.718149 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:18.737447 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.908342 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:18.910259 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:18.923586 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:18.965988 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:18.994680 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.000783 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:19.069373 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:19.108862 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:02:19.159418 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:19.188249 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240503 02:02:19.188859 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:19.219754 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.229450 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:19.290630 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:19.356812 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.406581 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:19.485342 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:19.496874 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.528856 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.566707 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:19.570988 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:19.577905 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:19.614784 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.642726 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:19.660298 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:02:19.695667 687 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:19.695802 687 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:19.696345 687 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:19.696749 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:02:19.696960 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:02:19.697335 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:19.697407 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:02:19.697486 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:19.697543 687 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:19.748116 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.769007 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:19.829517 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.848816 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:19.862680 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:19.886333 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:19.935094 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.942687 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:19.971910 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:19.982079 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:19.991533 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.004633 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.040444 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:20.042568 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.081837 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.115468 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.249691 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.263870 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.286875 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.287391 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.300186 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.344971 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.349236 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.394251 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.397086 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.422739 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.434613 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:20.438438 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:20.466542 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.468674 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:20.488972 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.491958 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.498926 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:20.540518 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:20.578766 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.646483 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.673358 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.726373 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.741504 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.776858 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.828779 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.829772 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:20.856539 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.953181 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:20.960542 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:20.989727 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:21.005738 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:21.017643 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:21.060534 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.118530 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:21.219962 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.226915 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:21.285800 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:21.303026 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:21.357688 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:21.361999 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:02:21.376415 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:21.463850 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.522672 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:02:21.523442 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:21.539436 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:21.561070 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:21.562602 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:21.616243 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:21.620647 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:02:21.662968 1115 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:02:21.663112 1115 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:21.663710 1115 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:21.664028 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:02:21.664183 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:02:21.664762 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.664883 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:21.664960 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:02:21.665092 1115 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:21.670583 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.715056 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.838523 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:21.869886 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:02:21.883368 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:21.900805 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:21.957581 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.020197 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.033694 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.068979 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:22.109215 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:22.120656 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.150053 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:22.260474 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.281843 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.296072 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.324031 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.341794 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.363798 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.399032 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.423063 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.434233 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.435412 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:22.525817 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.579468 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:22.593838 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.614259 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.636355 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:22.638680 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:22.653766 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.657258 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.667073 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.695304 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.696029 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.703826 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:22.793105 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:02:22.794363 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.821826 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.842852 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.891201 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:22.903769 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.924865 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.926442 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:22.946645 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:22.995725 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.008073 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.023337 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:23.083403 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:23.115592 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:23.118008 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:23.143352 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:23.189677 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.215855 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.255035 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:23.272362 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.276886 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.286139 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.329741 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:23.388111 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:23.501354 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:23.517408 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:02:23.563588 27840 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:23.563711 27840 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:23.564211 27840 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:23.564905 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:23.565075 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:23.565078 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:23.565188 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:23.565256 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:23.565428 27840 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:23.598035 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:23.635103 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.650411 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.662328 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:23.728101 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.783901 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.829681 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:23.848397 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:23.848462 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:23.879524 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:23.933316 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:23.961810 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:23.991973 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:24.027563 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.046828 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.048781 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:02:24.055742 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.106945 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:24.112354 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:24.180152 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:24.209596 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:02:24.267779 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:24.271453 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.400036 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.409107 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:24.441018 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:24.564718 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.591477 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:24.646854 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.663501 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.670060 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:24.694272 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:24.724838 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:24.737275 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.749174 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.839211 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.839576 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.889357 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.889357 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:24.915810 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.945672 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:24.988307 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:24.994686 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:24.998735 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.020200 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:25.061491 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.095407 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.097553 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.119200 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:25.156114 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.264253 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.291337 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.294242 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.348964 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:25.358533 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.412411 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.418174 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.422606 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.446882 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.474035 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.477218 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.486635 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.501482 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.516957 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.560103 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:25.565402 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:25.579658 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:25.609313 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:25.642313 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.761291 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.787766 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.821597 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.881639 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:25.891361 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:25.891747 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:25.910840 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:25.969996 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.063331 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.135378 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:26.176630 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:26.226482 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:26.227080 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:26.252952 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.257476 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.273288 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.322139 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:26.348239 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:26.463032 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.469111 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:02:26.473654 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.483155 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:26.502080 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:26.513835 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.517666 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:26.524878 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240503 02:02:26.578472 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.591364 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:26.677840 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:26.689844 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.767828 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:26.856832 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:26.880185 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.893765 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:02:26.909055 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:26.929352 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.076541 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:27.095433 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:27.139415 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.185564 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.186496 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.194141 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.196139 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:27.225200 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.239513 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.253026 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.286438 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:27.375993 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.377351 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.396163 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.429908 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.436156 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.556917 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.664947 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:27.672168 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.672587 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.686540 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:27.693918 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.694409 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.716786 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.749560 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:27.758021 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.774856 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.797962 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:02:27.812901 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.813647 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:27.827982 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:27.861652 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:27.890302 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.949898 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:27.962723 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.051364 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:28.097667 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.103879 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.126053 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:28.157441 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:28.163260 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.188234 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.205006 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:28.205806 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.257493 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.267468 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.268395 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.286387 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.288537 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:28.321508 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.379267 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.493919 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.520588 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.530191 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:28.701941 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:28.724282 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:02:28.739246 1795 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:02:28.739390 1795 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:28.739835 1795 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:28.740211 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:02:28.740361 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:02:28.740873 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:28.741304 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:28.741374 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:02:28.741533 1795 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:28.746794 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.786584 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:28.791477 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:28.792940 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.825448 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:28.908146 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:28.908712 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:28.929877 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.958829 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:28.965013 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:02:29.014837 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.037417 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:29.056464 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:29.137782 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.159988 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.172828 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:29.193558 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.215394 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.215580 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:02:29.230823 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:29.375648 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:29.426951 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.477329 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.520797 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.535122 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.556737 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:29.596565 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:29.686177 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.692076 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.712936 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:29.735373 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:29.744339 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.746553 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.872495 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:29.874408 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:29.896179 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:29.935829 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.957556 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:29.990012 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.002421 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.017285 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.036332 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.042968 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.125532 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.153280 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:30.178299 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:30.181638 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.237730 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:30.246529 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.270529 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.283562 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.320158 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.363948 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.373764 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.428400 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.439781 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.490927 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.501237 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:30.537930 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.567649 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.600695 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.610589 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.627452 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:30.630498 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:30.654495 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.661715 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.689857 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.727716 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:30.756486 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:30.793802 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.839964 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:30.850657 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:30.973731 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:30.976423 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.017261 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:31.077261 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:31.222172 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:31.229116 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:31.265774 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:31.318073 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:31.324164 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:31.328950 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:31.330405 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:31.353471 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:31.377210 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.379478 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:31.448979 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.480716 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.541555 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:31.571439 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:02:31.604903 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:31.642391 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:31.646409 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.683192 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:02:31.731158 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:31.757876 2472 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:31.758015 2472 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:31.758466 2472 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:31.758785 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:02:31.758956 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:02:31.759514 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:31.759626 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
W20240503 02:02:31.759721 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:31.759786 2472 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:31.811704 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:31.853139 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:31.863147 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:31.867132 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:31.904871 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:02:31.923980 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.969923 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:31.981432 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.009725 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.023602 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.081669 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.119092 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:32.165815 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:32.230312 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.260685 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.269749 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.364614 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.366158 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:32.367825 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.386868 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.405694 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:32.428648 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.462996 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.588704 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.594898 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.597690 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:32.656497 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.696957 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:32.706975 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.736160 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.748962 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.762642 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.789222 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.821906 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.836974 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.841243 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:02:32.847537 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.899811 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:32.938876 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:32.939435 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:32.980283 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:32.982414 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:33.030145 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.046985 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.065032 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:33.094609 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.099184 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:33.156666 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.160723 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.166000 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.261976 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:33.316903 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:33.328670 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.400087 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:33.410107 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:33.410100 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:33.491495 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.507325 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.549973 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:33.599593 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:33.666949 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:33.670815 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.699374 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:33.723762 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:33.725080 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:33.740154 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.818853 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.821516 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:33.843888 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.885478 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:33.914933 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:33.925123 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.929664 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:33.930603 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:33.945160 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:33.998471 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.021574 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:02:34.132891 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:34.163815 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:34.271950 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:02:34.272485 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.272619 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:34.334458 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.345446 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.348528 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:34.375360 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.378122 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.445422 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.464511 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.537578 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.577234 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:34.614181 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.677366 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.754272 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.755290 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:34.776119 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:34.828902 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:34.858104 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:34.868119 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.954046 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:34.958323 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:35.017395 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.020128 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:35.036712 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.069231 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.136235 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.147126 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.153496 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.187803 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.188987 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.217062 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.249084 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:35.249806 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:35.274960 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.351938 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.357897 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.413764 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.423877 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.469599 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.469681 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:35.476138 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.484231 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.511109 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:35.532948 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.544440 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.667893 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.698081 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.715914 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:35.716248 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:35.824586 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.828882 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:35.864691 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:35.933234 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:35.933786 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.982692 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:35.993957 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:36.050932 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:36.058559 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:36.110241 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.115686 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:36.116026 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.223336 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:36.228274 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.241899 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:36.259402 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:36.342779 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:36.365763 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:36.410007 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:36.435878 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.450410 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.466950 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.478991 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.491873 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.495074 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:36.512789 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:36.577373 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:02:36.592597 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:36.647874 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.703744 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.704736 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:02:36.727733 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:36.729992 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.768961 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.833777 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:36.854348 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.899731 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:36.949499 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:36.976267 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:02:36.978477 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:37.050479 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:37.060869 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.080395 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.132406 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.143631 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.269533 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:37.276041 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:37.335480 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:37.344564 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.387152 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.442940 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.468444 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:37.486699 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:37.488718 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:37.535113 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.549759 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:37.568593 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.569233 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.587493 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.596750 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.754971 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.809288 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:37.813546 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:37.823020 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.847070 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:37.874210 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.877411 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.898617 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:37.913225 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:02:37.916486 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:37.916568 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.023893 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:38.032967 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.050362 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.079630 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:38.100561 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:38.104153 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.119527 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:38.284667 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:38.307219 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.310050 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.331527 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.349041 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:38.356737 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:38.410460 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.505079 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:38.512951 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.561230 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.624529 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:38.652235 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.716176 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:38.759654 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:38.784947 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:38.786370 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:38.794314 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:38.804225 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:38.824211 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.830437 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.839228 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:38.870162 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:38.925437 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:38.940544 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:39.012153 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.013598 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:39.046774 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:39.062378 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:39.157194 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:39.162714 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:02:39.253765 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:02:39.271503 3316 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:02:39.271664 3316 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:39.272224 3316 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:39.272576 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:02:39.272728 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:02:39.273268 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:39.273375 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:39.273459 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:02:39.273599 3316 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:39.274295 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.288760 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:39.296613 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:02:39.297360 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:39.339591 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.380582 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:39.448232 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.523129 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.556609 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:39.591831 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:39.621397 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.632493 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:39.651158 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.656430 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.717063 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.728592 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:39.831082 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:39.862686 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:39.864449 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:39.910322 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:39.968798 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.024667 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:40.045791 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.048687 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.057080 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.058125 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.068735 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.071326 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.091411 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.145677 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.157596 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.196806 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.219780 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:40.264667 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:40.266470 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:40.322831 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.369124 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.408883 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:40.416045 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.426543 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.462554 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.506537 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:02:40.532666 3348 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:40.532821 3348 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:40.533416 3348 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:40.533720 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:40.533878 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:40.534372 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:40.534451 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:40.534512 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:40.534634 3348 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:40.539326 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.545096 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.627977 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.678882 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.692270 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:40.741264 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.770263 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.797968 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:40.821305 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.821741 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.825596 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:40.850257 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:40.903359 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:40.939182 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:40.945318 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:40.957757 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.094499 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:41.104650 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:41.105226 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:41.157301 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.243263 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.252832 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:41.260229 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.260514 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:41.286096 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:41.402997 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.409750 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:41.412771 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:41.460932 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.465857 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.494477 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.583340 7567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:02:41.645402 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:41.652598 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:41.667207 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.668767 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:41.686617 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:41.731524 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.766768 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.768286 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:41.836802 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:02:41.890986 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.893419 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:41.924669 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:41.990559 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:42.013831 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:02:42.066823 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.077035 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.088837 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.145931 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:42.213577 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:42.251731 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:42.271888 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.279091 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.287959 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:42.361196 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:42.362676 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:42.378576 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.403123 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.424925 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.486657 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.516487 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.551766 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:42.592041 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.593804 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.599303 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.608029 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.633126 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:42.656543 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:42.672966 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.700804 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:42.772717 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.860749 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.884058 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:42.914237 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 02:02:42.963070 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:42.963130 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:42.963809 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:42.986902 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.034580 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.107703 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:43.125193 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:43.140985 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.172777 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.179159 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:43.186054 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.213420 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.221684 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.257138 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.291952 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.348228 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.353781 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:43.422195 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.433933 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.485199 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.495389 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:43.497143 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:43.534144 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.549121 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:43.582096 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:43.673487 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.676862 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:43.688427 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:43.818775 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.836396 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.844393 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.844497 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.900825 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:43.946214 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:43.974630 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:44.022305 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.036990 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.044126 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.119073 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:44.161482 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:44.172096 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:44.226277 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:44.257539 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:02:44.307695 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.330826 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:44.339368 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:44.354022 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.372433 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.415903 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:44.433225 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.466995 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:02:44.491068 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.542586 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.565465 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:44.689965 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.708850 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:44.718226 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:44.747844 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:44.825321 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.840534 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.897814 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.899251 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:44.929622 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:44.942360 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:44.962626 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:44.989924 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:44.997056 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.011142 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.062469 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.073894 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.094521 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:45.122404 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.171744 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:45.181002 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.268414 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:45.270936 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.353293 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.362236 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.393819 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.446202 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.448253 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.463858 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.470419 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.515221 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:45.543522 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.570923 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:45.590315 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:45.661847 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.669647 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.716996 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.726410 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:45.734267 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:45.790107 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.827884 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.865286 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.883860 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:45.963068 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:45.979336 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:45.986207 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:45.986768 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:45.989017 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.017400 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:46.020959 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.036859 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.113310 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:46.247283 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:46.298976 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:46.309079 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.346745 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.363196 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.389561 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.431078 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:46.460548 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.468166 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.475634 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:46.488371 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:46.522171 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.543289 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:46.573024 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:46.583015 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:02:46.634363 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
I20240503 02:02:46.662307 3971 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:46.662453 3971 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:46.663116 3971 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:46.663532 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:02:46.663724 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:02:46.664279 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:46.664345 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:02:46.664618 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:46.664637 3971 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:46.670955 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:46.716130 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:46.745863 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.796545 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.809188 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:46.885442 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:46.889726 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:46.900583 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:46.931607 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.023780 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.030776 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.036386 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:02:47.036933 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:02:47.083858 3783 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:02:47.083997 3783 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:47.084446 3783 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:47.084822 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:02:47.084978 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:02:47.085438 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.085551 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:47.085603 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:02:47.085739 3783 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:47.093704 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:02:47.094321 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:47.107229 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:47.113134 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.170173 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.196166 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.304551 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:47.335075 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.338631 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:47.342939 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.378345 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.444324 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.445566 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.467772 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:47.483613 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.525018 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.559468 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.571727 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.611795 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.629772 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:47.639400 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.641168 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.643580 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.702411 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.735414 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:47.749261 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:47.804508 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.812760 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:47.854810 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:47.859612 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:47.882956 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:47.961613 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:02:47.982090 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.032961 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.035146 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:48.062561 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:48.072016 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:48.104745 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.120226 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.163311 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.227028 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:48.312243 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:48.327430 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:48.337070 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:48.358110 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.387100 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:48.430197 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:48.463649 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:48.464629 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.495182 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.517402 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.551057 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.614579 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.629405 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.667682 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.674278 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:48.698838 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.739615 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:48.764746 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.779040 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:48.858197 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:48.893785 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.896265 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:48.962134 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:48.984566 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.030218 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:49.039717 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.051769 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:49.052982 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.059437 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.090436 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.193140 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:49.196803 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:49.251823 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:02:49.332890 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.339094 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.342088 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.369525 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.373414 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:49.505165 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.523698 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:02:49.571702 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:49.572516 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.628288 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.637034 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:49.664914 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.687635 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.692214 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.746143 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:49.782397 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:49.842885 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:49.862807 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.879698 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:49.940394 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.947041 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:49.997665 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.032495 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:50.038789 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.040061 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:50.066506 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.105183 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.142405 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.183059 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.190277 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.208395 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.222661 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.240245 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.266785 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.291337 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.305192 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.327989 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.375964 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:50.436084 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:50.528858 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:50.565850 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:50.619298 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:50.628589 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.649778 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.652592 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.697474 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.758738 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.763856 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:50.842178 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.878595 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:50.882664 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:50.890875 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:50.956934 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:50.964874 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:50.984488 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.988236 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:50.997097 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:51.081096 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:51.156975 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.245800 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.279393 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.279945 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.343669 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.350241 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.363528 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:51.379050 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.379355 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.398526 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:51.406340 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:51.425171 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.453334 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.466203 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:51.487944 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:51.523500 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:51.571138 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.579262 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:02:51.596861 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:51.665746 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.670953 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:02:51.671619 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.728338 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.790218 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.820806 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:51.850319 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:51.851850 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:51.863404 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:51.881695 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:51.987654 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:52.041920 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:02:52.064285 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.115345 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.221655 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.245576 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.293908 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:52.307919 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.359974 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:52.364595 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.426218 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.447978 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.467867 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:52.495795 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.507946 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.514823 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.515439 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:52.542562 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.545466 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.595911 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.621615 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.727306 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.736418 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:52.752591 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:52.760784 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.834201 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.834619 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.844026 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.918220 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.924899 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:52.944118 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:52.962016 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:52.970914 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:52.984253 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.017437 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:02:53.028928 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:53.085078 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.085152 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:53.141414 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:53.143980 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:53.148614 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.174939 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:53.192761 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.223275 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:53.259066 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:53.374091 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.442797 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.448941 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:53.464704 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.488267 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:53.600813 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:53.613675 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.673367 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:53.713294 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:53.735272 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:53.773504 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.829281 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.859162 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:53.868477 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:53.870710 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:53.884649 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:53.979949 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.001082 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.057955 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:54.097880 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:54.108623 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.109697 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.128566 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.135068 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:54.135072 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.154261 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.156165 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:54.205827 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:02:54.207902 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.260455 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:54.293802 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.341984 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.380272 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.419790 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.498801 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:54.529495 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:54.576707 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:54.590850 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.594890 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.639155 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:02:54.639838 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:54.722918 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.725498 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:54.772315 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.807899 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:54.907610 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:54.977183 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:54.990646 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.000162 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.023108 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.083472 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:55.095362 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.105360 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:55.190176 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.197667 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.207775 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:55.223526 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.263758 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:55.338536 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.341869 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.386680 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.401436 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.417822 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.429280 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.454331 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.477188 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.480989 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:55.549371 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:55.572091 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.593698 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.618330 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:55.643380 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:55.647334 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.665622 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.745757 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.772467 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:55.818953 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:02:55.830786 5117 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:55.830986 5117 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:55.831648 5117 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:55.832338 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:55.832522 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:55.832859 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:55.832988 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:55.833048 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:55.833226 5117 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:55.833242 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:55.872771 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.933909 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.941349 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:55.981379 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:55.988616 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.023599 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.054733 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:56.076514 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.178268 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:56.185046 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:02:56.202813 5548 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:56.202972 5548 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:02:56.203527 5548 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:02:56.203967 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:02:56.204152 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:02:56.204591 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:02:56.204670 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
W20240503 02:02:56.204813 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:02:56.204838 5548 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:56.219218 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:56.294211 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.374461 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.410591 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:56.442616 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:56.528357 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.541565 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:56.541918 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.569712 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:56.570780 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.574043 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.597282 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.603686 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:02:56.648654 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.652540 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:56.655643 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.675824 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:56.702328 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.744149 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:02:56.758643 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:56.760874 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:56.914330 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:56.945948 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:56.978480 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:57.003403 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.034735 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.044867 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.100728 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.117853 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 02:02:57.150712 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:57.177467 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:57.184020 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.206044 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.209002 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.224941 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.289994 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:57.374011 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.378280 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.392218 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:57.480944 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.529024 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.647574 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.648648 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.649237 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.674569 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.711091 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:57.711094 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.720896 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.733652 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.764541 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:57.858242 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:57.863903 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:57.892498 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.897446 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:57.910158 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:57.927729 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.947122 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:57.992269 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.023535 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 02:02:58.063382 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.107095 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.125420 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.149960 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.224936 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.246181 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:58.253618 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.272215 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:58.282977 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.354707 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.370070 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:58.418378 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:58.444051 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:58.462108 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:58.464402 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.467226 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.488270 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.495436 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.537930 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.567901 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:58.679822 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:58.736665 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.737251 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.810019 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:58.911244 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:58.926666 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:02:58.953087 5117 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:02:58.953259 5117 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:02:58.953796 5117 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:02:58.954218 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:02:58.954401 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:02:58.955096 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.955605 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:02:58.955690 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:02:58.955830 5117 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:58.956308 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:58.973815 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:58.989042 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:58.995327 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.014935 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.076085 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.089671 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:59.123210 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:59.145579 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:59.150467 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.213459 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:59.244354 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:59.265995 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.336616 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:02:59.341674 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:59.343226 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:59.357707 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:59.394186 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:02:59.427843 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:59.503996 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.542026 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.595250 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:59.652168 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.680483 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:59.684772 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.685286 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:02:59.698282 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.699748 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:02:59.709163 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:59.721117 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:02:59.749157 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:02:59.774976 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:02:59.832830 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.876559 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:02:59.886029 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.012687 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:00.022472 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.068118 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.105019 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.110682 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.169401 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:00.188211 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.201675 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.218226 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:00.255040 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.262089 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.280534 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.321316 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.325142 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:00.366286 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.435310 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.441813 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.460582 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.506882 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.547003 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:00.585009 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.602103 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:00.611930 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:00.638844 6286 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:00.638962 6286 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:00.639461 6286 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:00.639801 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:03:00.639974 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:03:00.640547 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:00.640628 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:03:00.640730 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:00.640767 6286 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:00.699347 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:00.754586 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.788627 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.817385 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.827770 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:00.830602 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.887598 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.924599 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:00.928594 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:00.946787 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:00.978721 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:00.999845 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.022130 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:01.114665 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.115811 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.156924 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:01.165294 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:01.226987 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.228091 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:01.338855 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.394313 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:01.399530 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:01.401952 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.447665 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.449373 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:01.465943 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.468540 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.474269 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:01.502918 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.558661 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:01.559909 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.606597 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.624622 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:01.629382 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:01.660637 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:01.787523 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:03:01.796648 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 194 similar messages]
W20240503 02:03:01.797415 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:01.933169 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:01.939463 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.021914 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:02.024309 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.040841 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.050554 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.058743 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:02.064594 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.092479 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.117251 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.155149 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:03:02.155380 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.155895 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:03:02.204989 6441 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:03:02.205133 6441 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:02.205688 6441 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:02.206120 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:03:02.206281 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:03:02.206831 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:02.207274 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:02.207345 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:03:02.207484 6441 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:02.243173 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.277966 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.285275 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.301034 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.346904 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:02.400993 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.431007 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.448932 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:02.484280 5978 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:02.484431 5978 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:02.484947 5978 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:02.485433 7237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:03:02.485612 7237 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:03:02.486022 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.486109 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:02.486161 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:02.486325 5978 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:02.493875 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.514030 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.516337 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:02.518793 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.579430 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.593439 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:02.615926 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.703734 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.729130 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.771710 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:02.774652 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.798271 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:02.826118 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.858065 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:02.896549 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:02.994277 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:03.028614 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:03:03.039112 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:03.052393 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.078418 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.093377 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.093457 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.099047 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:03.109727 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.126674 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.133745 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.164014 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.188158 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.226069 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.303323 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:03.357558 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.360375 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:03.377501 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.427948 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:03.440639 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:03.520548 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.524123 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:03.534880 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:03.570605 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.679054 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:03.694119 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.712769 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.816999 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:03.843420 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:03.858750 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.883664 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.894960 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.927027 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.928735 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:03.969610 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:03.986318 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:04.062333 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.081501 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.097707 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:04.165901 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:04.203085 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.254952 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:04.367437 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:04.368357 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.369143 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:04.417912 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:04.419744 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.425938 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.438800 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.485517 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:03:04.584898 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:04.590322 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.612283 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.642733 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:04.697625 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.733243 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:04.756879 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.762750 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:03:04.819783 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.834877 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:04.885702 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:04.933068 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.953255 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.958459 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.967969 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:04.979223 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.036340 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:05.058347 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.102623 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:05.159396 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:05.182922 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:05.196062 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.209713 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.229072 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.231355 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.249563 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:05.305115 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.311144 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:05.441377 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:05.455788 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.502650 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:05.517372 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:05.541669 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.642272 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.651439 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.669932 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:05.695994 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:05.704753 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:05.724059 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:05.769529 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.776196 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:05.833323 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.866147 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.873533 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.893323 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:05.904577 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:05.914212 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:05.947535 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.948585 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.964692 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:05.999040 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:06.034461 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.064713 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:03:06.166618 6566 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:06.166790 6566 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:06.167341 6566 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:06.167862 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:03:06.168054 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:03:06.168574 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:06.168660 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:06.168743 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:06.168896 6566 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:06.197265 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.222311 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:06.263155 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.266400 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.282017 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:06.292651 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.327900 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.328840 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:06.385282 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:06.395067 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.431841 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.468689 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.506748 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.574612 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:06.648389 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:06.693284 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:06.698072 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:06.717504 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:03:06.739495 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.819707 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:03:06.832513 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.837347 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.869180 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:06.909363 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:06.954391 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:06.996358 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.006635 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:07.009837 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.031436 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.058509 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.116156 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.158659 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:07.168550 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:03:07.194320 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.198935 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.237994 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.291633 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:07.293864 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.331743 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.365252 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.412331 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:07.417716 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.499488 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.561882 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.581774 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:07.589246 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.589247 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.620014 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.627210 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:07.748965 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.802178 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:07.812873 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:07.821293 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:07.827425 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:07.855994 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:07.924485 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:07.972965 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.006448 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.020016 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:08.032094 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:08.040414 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.131114 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.134918 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:03:08.171916 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.177356 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.179445 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.301615 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.329551 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:08.359133 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.372438 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.384883 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.401034 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.405354 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.413918 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:08.436424 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.440889 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:08.459723 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.485167 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:08.547828 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.567813 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.592705 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:08.613343 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.741053 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.749192 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:08.770190 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.775218 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.806250 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:08.926196 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:08.933341 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:08.938552 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:08.958695 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:08.996347 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.038662 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.039285 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.049422 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:09.070168 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.078385 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.201733 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:09.208086 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.255967 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.318470 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.332494 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:03:09.358904 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.385296 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.399307 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:09.422374 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.464001 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:09.492846 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:09.579308 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.579897 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.599968 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.611830 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.620465 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.697780 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:09.728430 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:09.788301 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.798416 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:03:09.822731 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.854416 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:09.918576 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:09.946271 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:09.952900 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:09.981756 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:09.981760 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.034382 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.053303 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.133977 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:10.149942 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.181710 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.193665 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.197984 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:10.229759 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:10.291915 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.316563 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:10.333145 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.346535 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.350845 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:10.369879 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.379063 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:10.491799 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:10.538673 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:10.553993 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.651873 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:10.709395 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.714164 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.730294 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.731081 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.732332 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:10.766322 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.818818 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:10.837913 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:10.862465 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:10.868180 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:10.923453 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:10.985558 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:10.993047 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.107029 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.107650 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:11.115499 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.123365 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.146893 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:11.211139 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.248901 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.296481 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:11.443936 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:11.470496 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:11.486958 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.501958 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.533617 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.563238 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:11.594090 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.598520 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.600526 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.604772 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:11.610525 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.647486 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.731325 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:11.764990 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:11.828665 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:03:11.855994 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:11.885695 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:11.917742 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:03:11.922811 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:11.954486 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:12.037170 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.046612 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:12.060865 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:12.087370 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:12.116025 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.142999 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.155256 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.168677 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.196715 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.224320 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240503 02:03:12.232645 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:03:12.235626 8351 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:12.235764 8351 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:12.236411 8351 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:12.236912 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:03:12.237087 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:03:12.237648 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:12.237717 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
W20240503 02:03:12.237797 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:12.237864 8351 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:12.262418 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.292896 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.435539 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.443524 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:12.468428 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:12.503894 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.544255 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:12.545847 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.572106 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.576030 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.586517 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:12.601320 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.602350 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:12.669575 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:12.716853 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.773298 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.799786 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.820315 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.844813 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.890609 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.958415 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:12.961625 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:12.977203 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:12.992100 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:13.056890 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.100679 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:13.169103 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.170414 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:03:13.171023 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.175526 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.185243 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:13.208806 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.346920 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.349321 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.377434 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:13.391171 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.405663 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.406935 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:13.421272 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.428180 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.489478 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.558284 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.571751 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:13.581115 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.694737 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.699200 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:13.715574 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:13.744733 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.805262 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:13.840046 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.858911 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:13.950148 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.964643 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:13.968894 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.982132 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:13.994078 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:13.996402 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.077555 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.087834 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.101490 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.119179 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:14.164218 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:14.185333 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:14.194909 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:14.228206 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:03:14.355795 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.385627 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.393160 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:14.403869 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.480613 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:14.484969 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.606520 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.609148 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:14.616062 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.637576 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.653671 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:14.672511 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:14.695302 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:14.702389 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.789510 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:14.860940 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:14.908411 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:14.926739 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.962340 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:14.999068 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:15.003177 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:15.045696 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.049129 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.061276 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:15.081104 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:15.094135 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:03:15.094851 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:15.122009 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.128520 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.175755 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.257624 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:15.300871 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.369997 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.382781 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.404290 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.434191 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:15.442451 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:15.527446 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.572135 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.596628 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:15.611490 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:15.611594 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:15.615906 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:15.629299 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.650705 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.784926 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.787127 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.801802 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:15.855165 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:15.856839 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:15.890205 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.927055 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.965447 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:15.980911 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:15.993261 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.025581 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.026561 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:16.036770 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.131163 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.142973 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:16.176488 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:16.203914 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.283977 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:16.382815 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.424477 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:16.432027 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:16.460247 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.488565 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.495323 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:03:16.502295 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.502295 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:03:16.543769 8965 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:16.543903 8965 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:16.544435 8965 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:16.544796 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:03:16.544961 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:03:16.545521 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:16.545598 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:03:16.545686 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:16.545728 8965 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:16.553012 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:16.560287 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.569059 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.651291 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.665482 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:16.672523 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:16.724205 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:16.732441 6566 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:16.732596 6566 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:16.733176 6566 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:16.733582 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:03:16.733747 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:03:16.734241 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.734463 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:16.734516 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:16.734634 6566 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:16.739203 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.821508 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:16.869352 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:16.952353 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:16.977967 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:03:16.978657 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:16.983690 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.029330 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.069896 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.073153 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.109618 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.130404 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.163937 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:17.216540 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:17.233042 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:17.241858 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:03:17.243042 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.262640 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:17.379281 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:17.417436 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.434070 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.503602 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:03:17.505527 8903 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:03:17.505656 8903 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:17.506203 8903 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:17.506608 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:03:17.506760 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:03:17.507190 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.507275 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:17.507346 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:03:17.507473 8903 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:17.513795 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.517462 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.550206 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.572245 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.604671 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.616628 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:17.641189 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.647840 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.664708 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:17.738862 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:17.743618 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:17.762668 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:17.857144 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.901254 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.957630 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:17.971741 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:17.995857 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.087982 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.095307 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.109474 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.111917 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.151697 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:18.170164 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.177886 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:18.182826 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:18.239629 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:18.246199 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:03:18.246817 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.250085 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.292587 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:18.300910 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.332206 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.337566 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.363406 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.391973 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.441457 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.513644 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.526724 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.639638 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.639799 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.679378 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:03:18.681350 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:18.707751 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:18.749284 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.797319 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:18.854840 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:18.889695 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.892694 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:18.931116 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.947165 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:18.980139 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.051766 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.095209 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:19.098754 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:19.105046 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:19.235606 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.247439 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:19.252185 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.270622 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:19.306998 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:19.309859 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:19.350149 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:19.390211 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.406065 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:19.452339 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.466770 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.486582 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.526054 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:19.531327 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:19.608156 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:19.628693 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:19.704782 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:03:19.780669 9246 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:19.780822 9246 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:19.781319 9246 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:19.782054 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:03:19.782241 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:03:19.782778 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:19.782881 7568 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, bed387c0ff9241688cdf45f364dc6f5a
W20240503 02:03:19.783090 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:19.783298 9246 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:19.810973 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:19.827947 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:19.852885 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:19.871992 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:19.878496 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:19.989710 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.006167 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.009471 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.009567 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.052811 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.056092 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.093537 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:20.117681 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.132553 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.187682 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.226717 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 02:03:20.227559 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:20.253252 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:20.272882 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.344539 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:20.396611 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.430799 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.447939 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.493706 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.502552 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.527825 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.565917 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.574170 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:20.635669 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.654507 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.663215 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.671643 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:20.688488 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.693389 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.719461 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:20.726418 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.765607 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:20.780714 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:20.817559 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.823326 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:20.846621 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:20.890177 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.975868 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:20.984526 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.014564 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.108549 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.165282 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:21.172084 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:21.242481 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:21.246670 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.283649 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:03:21.319440 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.322937 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.324837 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:21.463572 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.494616 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.542169 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:21.551683 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:21.562242 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.605463 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.627107 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.630563 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:21.674278 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.727325 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.749506 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.763239 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:21.771626 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:21.776540 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:21.852811 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:21.927281 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:21.995481 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.003970 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 194 similar messages]
W20240503 02:03:22.066249 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:22.089200 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.098008 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:22.105612 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.145184 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.161813 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.197610 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:22.203565 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.204569 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.228085 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.232160 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.258816 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:22.280898 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:03:22.421366 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.488122 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.503731 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:22.524258 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.576212 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.601621 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:22.626587 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.635949 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:22.687860 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.718089 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.723899 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.727504 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:22.740726 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.745111 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.797500 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.817668 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:22.915570 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:22.916034 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:22.954413 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:22.981155 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.046916 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:23.107774 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.108151 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:23.159648 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.166069 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:23.181660 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.189356 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.198011 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:23.227210 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.252105 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.252847 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:03:23.256121 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.264742 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.329993 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.343513 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.378890 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.389264 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.427448 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:23.497776 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:23.518858 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.547466 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.589601 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.596108 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.665846 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.685704 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.735736 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:23.771030 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.773072 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:23.773803 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:23.814218 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:23.844029 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:03:23.846706 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:23.898397 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:24.051323 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:24.069159 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.109688 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.132500 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.170467 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:24.170547 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.180888 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.191901 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.260193 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:24.282408 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:24.313261 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.321290 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:24.368569 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.394790 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.513832 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:24.558091 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:24.578091 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:24.595273 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.622535 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.644935 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.699965 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:24.701097 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:24.725736 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:24.728399 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:24.754081 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:24.767318 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:24.824981 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:24.853333 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.036453 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.037954 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.099723 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.101615 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.119936 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.150072 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.159309 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:25.167727 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:25.202638 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:25.205739 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.260730 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:03:25.261492 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.329375 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.353276 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.364826 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.400527 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.414019 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.419190 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.420392 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:25.439772 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.498185 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.508826 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.548815 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.559595 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.588840 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:25.671476 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.681874 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.692373 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:25.709693 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.720173 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.750538 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:25.759030 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.767844 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:25.802709 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:25.814277 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:25.816684 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.935604 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:25.978559 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:26.006059 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:26.039201 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:26.041103 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.126700 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:26.175776 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.240937 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.278652 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:26.291188 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.310580 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.319372 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:26.333274 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.385202 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:26.422076 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.437518 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:26.502166 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:03:26.550266 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.567874 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.621187 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.654754 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:26.670568 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.671759 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:26.723001 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.742784 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.798666 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.825749 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:26.885771 9624 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:03:26.885927 9624 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:26.886458 9624 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:26.886927 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:03:26.887099 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:03:26.887831 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:26.887919 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:26.888005 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:03:26.888136 9624 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:26.922242 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:26.928947 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:27.002943 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.012585 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:03:27.013350 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.046633 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.111352 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.126129 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:27.138613 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.167166 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.193413 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.225632 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.227128 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:27.232399 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:27.239398 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:27.304011 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:03:27.304802 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.320748 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.371520 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.447917 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.551869 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:27.593106 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:27.631381 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:27.670924 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.710336 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:27.724511 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.736176 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.739303 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.741407 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.782867 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.793349 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.825109 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:27.911039 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:27.926129 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.928649 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:27.963791 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.969271 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:27.970820 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:27.998703 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.019920 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.032150 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:28.095175 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.113726 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:28.127238 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.146301 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.171370 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.183084 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.210729 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.252585 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:28.276955 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 02:03:28.375860 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.399340 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.424894 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:28.455080 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:28.471602 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.471602 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.482010 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.500836 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.502671 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:28.546254 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.589013 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:28.599748 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.631032 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.679831 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.705399 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.830797 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.839033 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:28.876627 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:03:28.895118 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:28.902994 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:28.935390 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:28.948266 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:29.002108 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:29.056404 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.098655 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:29.135015 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.273618 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.279019 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:29.287497 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.340608 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:29.372849 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:29.384912 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:29.398128 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.459501 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:29.486383 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:29.514281 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:29.521741 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:29.533325 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.634027 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:29.692662 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:29.702489 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:29.743162 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:29.792202 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:29.803249 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.823727 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:29.883507 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:29.924607 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:29.968700 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:29.992663 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:30.031489 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.066300 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:30.077994 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.081624 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:03:30.086182 10208 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:30.086309 10208 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:30.086827 10208 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:30.087186 7237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:03:30.087368 7237 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:03:30.087877 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.087981 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:30.088048 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:30.088176 10208 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:30.095454 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.118479 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.120872 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.286711 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:30.287734 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.340267 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.381950 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:30.388629 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.391062 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.391100 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:30.409633 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.416910 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:03:30.429966 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.480733 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.533663 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:30.540778 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.553120 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:30.560817 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.577788 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.636608 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.661353 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.683147 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.728617 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:30.748021 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.789966 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:30.790050 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:30.822576 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:30.838443 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:30.858373 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.877291 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.952629 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:30.966795 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:30.985090 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.039163 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.049158 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:31.073135 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:31.073458 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.084072 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:31.087914 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.099419 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.216784 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.222414 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:31.349650 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.352862 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:31.381295 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.409574 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:31.473367 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:31.478004 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:03:31.604497 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.623989 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:31.637981 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.656219 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:31.695781 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.730253 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.748826 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:31.759902 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:31.803241 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:31.858263 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:31.877427 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:32.016594 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:03:32.042660 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.055029 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.160692 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.171356 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.191821 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.198341 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:32.210803 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.216151 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.255503 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.266171 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:32.266417 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.317116 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:03:32.344276 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:32.365190 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:32.433457 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.536607 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.620157 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:32.635046 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.638432 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.646858 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.656474 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:32.670157 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.686682 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.709753 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.717646 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:32.724524 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:32.763175 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.763589 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:32.773252 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.801877 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.880676 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.886158 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:32.914949 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:32.935331 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:32.943225 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:32.991564 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.143935 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:33.179272 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.186208 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:33.192335 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:33.201345 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:33.208420 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.213078 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:33.236063 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.286494 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.287494 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.311429 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:33.313284 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:33.323118 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:03:33.372334 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.398108 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.494580 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.503100 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:33.526142 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.621081 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:33.623379 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.672596 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.679826 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:33.742080 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:33.743175 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:33.753716 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:33.754205 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.760794 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.766163 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.872757 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.876607 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:33.889603 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:33.892146 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.058187 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:34.094650 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:03:34.115025 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:34.121055 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.165410 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:34.256484 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:03:34.268177 11293 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:34.268319 11293 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:34.268805 11293 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:34.269162 7237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:03:34.269318 7237 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:03:34.269765 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:34.270165 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:34.270224 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:34.270381 11293 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:34.271919 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:34.277892 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.287940 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:34.320286 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.357527 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:34.375478 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:34.463754 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:34.619010 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:34.629709 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:03:34.667162 10955 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:34.667315 10955 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:34.667941 10955 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:34.668366 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:03:34.668584 7370 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:03:34.669004 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:34.669076 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:03:34.669190 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:34.669237 10955 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:34.694077 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:34.705008 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.749902 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.749919 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:34.753815 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:34.763453 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:34.806677 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:34.819132 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:34.883288 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:34.898046 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.014330 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:35.036662 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.065353 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.151226 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.153501 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.195258 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:35.198416 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.205703 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.218556 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:35.232195 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.258366 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.271003 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.291419 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:35.312680 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.339900 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.341926 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.344218 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:35.353484 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.366052 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.372077 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.417068 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:35.606180 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.632378 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:03:35.651466 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.676016 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.677058 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.680344 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:35.688581 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:03:35.708070 11486 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:03:35.708209 11486 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:35.708657 11486 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:35.709075 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:03:35.709224 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:03:35.709628 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.709712 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:03:35.709795 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:03:35.709944 11486 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:35.737535 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.756410 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:35.760541 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:35.763883 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:35.788144 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:35.842397 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:35.863829 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:35.914511 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:36.033314 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:36.067282 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:36.069103 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.100796 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:36.160887 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.223680 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.224464 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.252187 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:36.258435 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.283268 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:36.294490 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:36.296453 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.314004 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.369589 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.384387 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:36.412019 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:36.413148 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.511166 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:03:36.556475 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.569633 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.626152 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.632483 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.754401 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:36.779362 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:36.795553 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:36.797297 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:36.830670 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:36.883788 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:36.913247 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:36.939567 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:36.969273 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.025127 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:03:37.095880 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.124372 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.143285 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.166105 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.234122 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.244864 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.267436 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:37.327380 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:03:37.328090 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.343513 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.364034 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:37.395278 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:37.457788 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.463501 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:37.508343 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.522300 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.524937 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:37.546200 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.569155 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.596642 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.670899 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.682338 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:37.720954 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.722271 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.734726 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:37.816547 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:37.832397 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:37.860594 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.870553 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:37.881191 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.901458 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:37.968178 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:38.048012 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:38.051503 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.083971 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.096503 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:38.104951 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.115176 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.153290 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.182832 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:38.195641 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.199586 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.225903 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.348349 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:38.366250 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 02:03:38.408259 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.411485 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:38.430841 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.436682 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.471427 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.562754 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:38.566684 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.576243 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.582497 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.604151 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.627699 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:38.664960 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.732126 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:38.746335 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.794039 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:38.847954 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.848891 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:38.937932 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.938475 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:38.952570 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:38.955780 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:39.020979 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.037014 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:39.071457 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:03:39.219856 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:39.225930 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.239286 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:39.265800 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.358915 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:39.373790 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:39.400910 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:39.537128 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:39.540431 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.557102 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:39.577430 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:39.596390 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:39.607646 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:39.683768 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.701308 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:39.746569 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.749794 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:39.807042 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:39.914968 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:39.969887 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:39.971028 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:39.978029 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:39.995019 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:39.996151 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.020349 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.033833 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.095203 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.098240 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.116564 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.123865 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.137655 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:40.163022 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.165161 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:40.199514 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.280331 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.286388 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.305493 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.345099 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:40.421738 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.431334 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:40.433660 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.487128 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.518644 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:40.532873 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.544725 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.547610 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:40.549700 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.618077 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:40.633637 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.661263 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.693698 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:03:40.771584 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:40.801635 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:40.874759 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:40.898733 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.910708 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:40.915588 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:40.937759 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:41.019184 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.034914 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:41.066892 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.073844 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.101215 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.116129 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.140024 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:41.164099 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:41.164578 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:41.183120 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.217078 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:41.296691 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.358000 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.361882 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:41.506886 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:41.560526 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:41.577735 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.593417 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.597168 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:41.651243 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:03:41.664263 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.676801 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.696364 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:41.769747 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:41.817541 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:41.840682 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.870000 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:41.985551 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:41.991072 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.032217 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:03:42.075162 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.089430 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.103111 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.127039 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:42.241081 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.264636 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.304482 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:42.310694 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:42.314906 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.327564 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:03:42.356746 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.372638 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.389436 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.393242 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.441473 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.461715 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.470782 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:42.507454 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.511648 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:42.530930 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.564702 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.575778 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:42.576783 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.584165 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:42.657778 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.692366 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.738781 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.774099 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:42.797621 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:42.809309 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.817566 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:42.899513 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:42.934701 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:42.954865 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:42.967222 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:42.998450 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.001674 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.058324 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.105973 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:43.182796 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:43.209578 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.210704 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.224421 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.224442 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.231474 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.239228 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:43.240757 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:43.265262 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.272776 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.337625 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:43.367267 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:43.467648 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:03:43.468080 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.529789 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.546640 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.547703 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:43.556936 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:43.580631 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.585119 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:43.690788 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.715982 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:43.718991 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:43.732589 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:43.747571 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.842738 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:43.860240 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:43.875053 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:43.949988 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:43.985769 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.080222 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.115772 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.179637 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:44.184516 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:03:44.217777 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.263947 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:44.311582 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:44.331952 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.337575 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.398234 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:44.489176 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:44.524717 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.535982 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.563162 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.701385 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.707105 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:44.714870 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:44.717988 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.812770 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:44.862393 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:44.867599 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.871619 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:44.927000 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:44.947496 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:44.965658 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.006299 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:45.084297 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.098006 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:45.102036 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.105818 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.120524 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:45.129336 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.140297 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.166973 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.202860 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:45.256707 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.332010 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:45.332790 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:45.362512 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.367586 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:45.405797 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.433744 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.441208 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.447701 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.481762 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.531883 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.558890 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.567410 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:45.627197 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:45.655121 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.691565 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.694128 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:03:45.694720 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.696264 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.739156 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.740408 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.787168 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.812492 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:45.841012 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:45.888190 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:45.908187 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:45.968801 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.975143 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:45.993955 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.048079 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:46.081564 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.135185 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:46.155865 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:46.221297 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.234881 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.255184 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:46.327832 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.358973 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.398278 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.439611 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.439682 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.444795 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:46.469796 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:46.500125 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:46.551185 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:46.581440 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:03:46.672639 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:46.758940 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.850222 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:46.899629 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.916438 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:46.943370 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:46.999848 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.037856 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:47.038344 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:03:47.038962 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.055977 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:47.117218 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.143690 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:03:47.144433 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 97ffb59d3ab64d03aca5d30a7c3be1e1: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:47.213367 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.253942 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:47.331748 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:47.340011 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.363754 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 02:03:47.364360 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.392848 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.400986 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:47.403203 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.436321 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:47.437537 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:47.438236 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:47.440798 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:03:47.491916 12915 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.492070 12915 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:47.492638 12915 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
W20240503 02:03:47.493285 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:47.493274 7237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:03:47.493443 7237 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:03:47.493723 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:47.493806 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:47.493891 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:47.494028 12915 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:47.497972 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.514678 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:47.516224 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:47.564142 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.582103 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.651942 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.682133 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.687089 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.703626 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:47.741715 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.756392 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:47.809752 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.836503 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:47.853009 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:47.871140 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:47.881237 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:47.932664 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:47.963891 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:48.016685 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:48.019052 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:48.028569 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.079504 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:48.086804 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.105988 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.221657 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.223119 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.243846 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:48.300714 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:48.320139 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.363571 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.373520 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.380856 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:48.394208 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:48.402001 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:48.482718 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:03:48.484189 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.497009 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.555024 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.566872 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:48.627818 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:48.631750 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:48.652619 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.693398 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:48.759768 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.800202 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.801641 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:48.875669 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:48.913298 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:48.931778 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:48.966774 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.016453 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.066504 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.094143 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:49.109582 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:03:49.130774 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:49.178637 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:49.203078 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:49.292982 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.339314 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:49.398120 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.428459 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.456523 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.599910 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.624723 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:49.643957 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:49.671145 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.679266 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.697621 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.708952 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:49.717774 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.805473 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:49.847824 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.851706 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:49.858104 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:49.865434 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.905970 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:49.926291 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:49.945871 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.000015 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:50.010921 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.054683 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:50.058673 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.063184 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:50.069955 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.124490 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.193356 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.210489 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.216675 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:50.251775 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:50.281600 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.298918 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.340349 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.356806 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.366878 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:50.366952 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:50.416004 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.449054 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.450273 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:50.490415 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.550567 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:50.568722 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.615701 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:50.676514 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:50.772313 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:50.858765 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:50.864449 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.892503 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:50.904542 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:03:50.905202 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:50.928750 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:50.943997 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.953699 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:50.966847 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.058158 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.084195 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:51.112820 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:51.116034 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.144023 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:51.193255 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.226649 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:51.262251 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:51.299855 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:51.313727 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.351264 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.401408 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:51.441228 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.443303 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.516764 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.521582 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.545549 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:51.671454 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:03:51.676723 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:51.748374 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.759655 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:51.781646 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:51.791785 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:51.835253 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:51.839578 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.890699 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:51.931442 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:51.971907 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.014286 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.044755 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 187 similar messages]
W20240503 02:03:52.097046 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:52.165959 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.194906 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.199342 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:52.217690 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:52.217819 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.241151 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.247241 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:52.270529 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.275591 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.345120 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:52.373203 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 02:03:52.373852 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.445500 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:03:52.469009 13399 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:03:52.469146 13399 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:03:52.469753 13399 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:03:52.470299 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:03:52.470448 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:03:52.471232 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:52.471432 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.471429 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:03:52.471489 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:03:52.471652 13399 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:52.493075 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.556687 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:52.560150 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.594882 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.623641 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.628721 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.629875 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:52.652875 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.662835 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:52.764480 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:52.775631 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.787258 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.809100 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:52.841816 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.856473 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.859670 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:52.881036 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:52.890899 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:52.948094 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.012185 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.035858 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.047884 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:53.070827 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:53.109022 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.196056 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.281719 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.294836 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.301884 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:53.304875 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.347000 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.347589 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.362762 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:53.381052 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:53.480479 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.502171 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 02:03:53.502805 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.510187 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.522167 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:53.580907 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:53.606280 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.637948 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:53.720647 13430 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:53.720800 13430 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:53.721385 13430 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:53.721850 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:03:53.722019 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:03:53.722505 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.722594 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:03:53.722735 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:53.722937 13430 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:53.725817 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.758555 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:53.761791 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.842733 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.875345 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.881191 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:53.900014 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:53.950172 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:53.981613 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:54.012466 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:54.030699 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:03:54.078646 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.104723 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.118145 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:54.152019 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.227073 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.290750 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.338140 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:54.416702 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:54.417207 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.441794 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:54.442824 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.445994 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.516242 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:54.521423 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:54.563122 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.568346 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.603463 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:54.637238 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.638465 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.700196 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.801122 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:54.817891 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:54.822687 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:54.824882 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.846060 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:54.914286 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:54.968183 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:54.996181 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.049918 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.050588 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:55.057415 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.081353 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:55.092368 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.173708 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:55.267767 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.283248 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.295894 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.313120 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.314011 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.323796 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.362129 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.383834 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:55.462273 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:55.462476 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:55.467285 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.481456 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.498356 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.559199 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.670642 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.672226 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:55.711814 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.748464 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.776798 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.785884 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.808123 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:55.812707 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:55.832345 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:55.845192 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:55.874380 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:03:55.933647 13430 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:55.933808 13430 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:03:55.934377 13430 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:03:55.934804 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
W20240503 02:03:55.934888 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240503 02:03:55.934984 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:03:55.935566 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:55.935663 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:03:55.935735 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:03:55.935874 13430 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:55.958676 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:55.964635 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:55.969303 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:56.063472 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:56.102481 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.104164 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.152535 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:56.152834 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.183151 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:56.214147 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.230506 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:56.236671 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:56.271261 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:56.307052 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:56.398219 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.421762 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.463510 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.510191 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:03:56.524288 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.572434 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.573805 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:56.646272 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.706920 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.779924 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.819670 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:56.831924 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.843277 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:56.853670 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:56.869434 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:56.892339 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:56.895972 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:56.917949 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:56.986039 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.069841 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:03:57.070569 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.076046 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.129016 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.152516 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:57.155764 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:03:57.237900 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:57.250627 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.264626 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.267095 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:03:57.276588 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:57.335083 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.386102 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:03:57.418906 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:57.450757 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.452634 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.474630 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:57.549121 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.570020 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.676625 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:57.690052 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:57.701676 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:57.732748 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.786332 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:57.803725 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:57.836120 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:57.841734 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.943888 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:57.943913 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:57.947250 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:57.948609 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.978771 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:57.988240 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.051934 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.099078 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:58.109941 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:58.141999 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.144062 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.205320 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:58.209821 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.260162 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:03:58.288548 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.373392 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.384048 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.391822 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.417294 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:58.430800 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.464601 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:58.492880 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.512969 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:58.549393 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:58.555565 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:58.582840 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.593624 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.633276 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.640204 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.645146 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:03:58.692859 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.700242 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.719810 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:58.748001 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:58.927507 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:58.930330 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:58.934357 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:58.957450 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:03:58.977421 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:59.033279 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.035033 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:03:59.098115 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.121191 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:59.201783 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.206665 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.337440 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.355392 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.361511 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:59.375558 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.394263 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:03:59.414119 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.422379 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:59.474179 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.548188 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.561439 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:03:59.569072 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.574733 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.648319 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:03:59.675462 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.728919 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:59.757189 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:03:59.775830 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:03:59.791281 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:03:59.864449 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.871507 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:03:59.934631 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:03:59.997020 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.013204 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.030390 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.051195 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.105396 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.115363 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.163856 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:00.167953 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.170181 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:00.254241 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:00.261684 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.405275 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:00.412199 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.418699 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.439764 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.445820 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.446424 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.488824 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.585796 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.602245 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:00.635605 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:00.638729 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.650820 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:00.658782 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.704361 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:00.779055 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.780573 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.817016 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.832566 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:00.836203 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:00.853240 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.885525 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:00.948963 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:00.988085 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.008356 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.075271 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:04:01.093271 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.102239 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:01.115294 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.141902 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.153626 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.164418 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.196578 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:01.235777 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.238276 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:01.243955 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:01.268801 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.359946 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.373363 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.377887 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.427021 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.450696 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.470503 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:04:01.615612 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.648483 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:01.684765 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:01.692487 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:01.756724 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.785092 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:01.802991 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.806095 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:01.838629 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.847365 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.856667 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:01.877911 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:01.881217 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:01.962414 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:01.998903 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.026285 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.069485 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:02.073091 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.169170 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 191 similar messages]
W20240503 02:04:02.221186 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:02.256160 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:02.293697 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.325858 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.339000 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.376997 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.389586 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:02.398425 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:02.403797 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 02:04:02.483506 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.519419 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.582306 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.586377 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.618140 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:02.672900 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.718036 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:02.796057 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:02.811100 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:02.811290 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.858027 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.867074 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.886445 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.886445 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.886579 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:02.894924 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:02.949478 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:03.024081 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.047688 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.121917 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:03.123857 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.135007 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.164305 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:03.214782 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.236215 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:03.246960 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.255621 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:03.329434 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:03.352399 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.352422 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.369320 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:03.377100 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.384837 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.415642 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.494292 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.561818 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:03.566349 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.608943 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.631008 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.705555 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:03.705682 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:04:03.714706 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.721035 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:03.746052 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:03.761497 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:03.762409 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.785671 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:03.841545 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:03.855702 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:03.909114 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:03.910494 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:03.923723 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.044793 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:04:04.048103 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.067358 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.122063 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:04.198307 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.231937 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:04.241772 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:04.258739 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.272595 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:04.292577 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:04.300561 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:04.320272 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:04.342442 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.344398 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.421875 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.446372 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.534147 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.564985 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.630784 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.687691 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:04.728574 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:04.736713 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:04.781198 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:04.793368 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:04.804525 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:04.829510 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.917501 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:04.933537 15188 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:04:04.933656 15188 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:04.934181 15188 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:04.934495 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:04:04.934635 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:04:04.935184 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:04.935273 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:04.935334 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:04.935439 15188 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:04.991884 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.007038 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.063449 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.109458 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.120882 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:05.129016 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:05.212699 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:05.249794 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:05.287468 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.303184 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:05.323621 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:05.370918 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.377689 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.511952 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.538657 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:05.544867 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.573982 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.630234 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:05.643507 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.649561 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:05.659006 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.678558 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:05.685940 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:05.706815 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:05.760360 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.762948 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:05.792119 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.805363 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.904815 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:05.917388 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.918862 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:05.940452 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:05.992704 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.023154 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.032480 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.071810 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.091434 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:06.103469 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:04:06.104182 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:06.108043 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.119387 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.123625 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.134214 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.145655 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.166359 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.268512 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:04:06.277510 13430 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:06.277667 13430 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:06.278235 13430 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:06.278558 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:04:06.278720 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:04:06.279389 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:06.279472 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:06.279529 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:06.279691 13430 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:06.280391 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:06.316689 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:06.338606 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.346989 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.381739 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:04:06.401144 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.480798 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.493166 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:06.516950 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.556480 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:06.588660 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.604748 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:06.663359 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.785194 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:06.795855 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:06.804355 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.883893 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.920497 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.924638 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.934089 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:06.986608 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:06.991515 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.070832 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:07.077361 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.078778 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:07.096563 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:07.105926 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:07.172761 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:04:07.227205 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:07.258111 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:07.304150 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.341424 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:07.373215 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:07.407733 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.425401 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.425590 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:04:07.447818 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.555859 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:07.602538 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.626924 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:07.669364 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.786547 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:07.807590 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.818068 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:07.840005 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:07.860344 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.876673 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:07.881904 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:07.933897 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:07.951619 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:07.972940 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.013355 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:04:08.029089 13430 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:08.029218 13430 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:08.029666 13430 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:08.030019 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:04:08.030200 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:04:08.030726 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:08.030864 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:08.030939 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:08.031052 13430 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:08.160795 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.185777 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.188498 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.203787 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:08.247506 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:08.253998 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.318951 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.329219 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.337020 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.352811 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.358426 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:08.371150 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:08.384091 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.414762 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.420197 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:08.424938 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.430922 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.456862 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.467579 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.481595 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.508860 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.576913 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.579355 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.611162 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:08.653970 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.671330 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:08.683138 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.732085 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:04:08.738943 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.884948 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:08.894013 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:08.908373 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:08.945633 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:08.951635 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:08.965597 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:09.017205 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.067294 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:04:09.099320 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.100013 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.130613 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.164577 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.192667 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:09.210934 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:09.222592 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.244902 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.317724 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.327229 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.356462 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.399583 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:09.414090 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:09.420397 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:09.510746 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.544545 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:09.547479 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.594805 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:09.639899 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.701982 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:09.708848 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.739231 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:09.803215 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:09.819780 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:09.843922 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:09.897130 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:09.901827 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:09.913297 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:10.033233 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:10.062978 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:10.069341 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:10.086184 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.199833 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.236927 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.238761 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:10.276306 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.278928 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:10.405928 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:10.460366 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:10.474278 16022 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:10.474488 16022 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:10.475023 16022 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:10.475328 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:04:10.475507 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:04:10.476168 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:10.476241 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:10.476370 16022 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:10.476390 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:10.488997 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:10.496883 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.498656 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:10.580550 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.632472 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:10.657336 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.686743 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.686853 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.708803 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.793543 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.819094 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.849289 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.876226 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.886931 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:10.911679 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.920186 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.940063 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:10.953439 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.956822 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:10.962530 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.974006 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:10.987087 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:10.994699 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:11.002928 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:11.024374 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.043300 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.065706 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:11.081466 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:11.109239 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:11.183813 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.209856 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.283994 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:11.297926 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.350072 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:11.353142 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.393433 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 02:04:11.459408 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.481253 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.492895 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:11.546185 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:11.547147 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.608587 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:11.684067 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:11.750861 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.772473 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.807040 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.817143 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:04:11.822664 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:11.861687 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.880256 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.908707 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:11.935940 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:11.967293 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:11.973803 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:12.013585 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:12.170759 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:12.173563 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240503 02:04:12.174127 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.185564 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:12.192044 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:12.250000 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.260627 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.263784 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:12.263837 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:12.302341 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:12.319402 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:12.351614 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:12.397084 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:12.405366 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:12.413488 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:12.456598 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:04:12.600963 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.622721 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:12.728180 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:12.770233 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.792335 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:12.812409 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:12.814276 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:12.818732 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.011921 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:13.021773 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:13.064165 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.102146 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.137681 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.141160 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.150902 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.158324 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.176697 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.205686 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.223691 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.226140 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.245900 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:13.248796 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.300019 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:13.319094 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.333051 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.337512 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.357237 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.379251 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.436952 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.460786 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:13.480239 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.485980 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:13.516659 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.524971 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.541121 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.576207 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:13.588547 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:13.605746 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.696382 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:13.742240 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.771131 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 02:04:13.771857 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:13.784644 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:13.805342 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.833108 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.840703 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:13.841300 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:13.931200 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:14.027880 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:14.038089 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.041447 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.050081 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:14.153172 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:14.201364 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.273988 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.283520 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.385270 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.413409 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.442729 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:04:14.447888 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:14.459616 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.462095 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:14.476014 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.495846 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.499465 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:14.533334 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:14.577785 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:14.592404 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:14.599315 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:14.759147 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:14.781702 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:04:14.805604 16518 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:14.805742 16518 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:14.806293 16518 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:14.806820 7502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:04:14.807008 7502 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:04:14.807425 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:14.807502 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:14.807610 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:14.807741 16518 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:14.826280 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.863848 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:14.864629 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:14.866153 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:14.932098 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:14.958426 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.030156 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:15.039703 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:15.109306 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:15.109923 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:15.137990 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:15.151453 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:15.241068 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.317047 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.410220 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.414817 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.482125 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:15.488523 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:15.497310 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.497714 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:15.557224 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.579003 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:15.598574 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:15.605618 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:15.627754 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.631423 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.646389 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.685834 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.703264 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.722885 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.772454 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.782433 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:15.786553 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.802717 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.827323 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.873891 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:15.934013 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:15.961570 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.001243 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:16.010893 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:16.012045 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.035719 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:16.056895 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.108573 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:16.143607 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.185048 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:16.218284 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:16.231627 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:16.239703 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.260872 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.288476 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.354097 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.419891 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240503 02:04:16.435220 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.491315 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.577925 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:16.578514 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:16.611727 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:16.628212 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:16.647167 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:16.659107 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.682519 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.693176 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:16.736719 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.838686 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:16.868510 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.876161 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.882323 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.883595 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:16.928210 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.960165 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:16.998111 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:17.028333 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:17.072913 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:17.085724 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:17.085959 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:17.178201 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:04:17.190766 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 190 similar messages]
W20240503 02:04:17.205785 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:17.206588 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:17.281374 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:17.285619 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.291603 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:17.387104 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:17.425864 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.468005 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.505787 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:04:17.546550 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:17.599356 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:17.612727 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:17.688458 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:17.809435 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:17.823587 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:17.837741 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:17.863032 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.863121 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.884446 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:04:17.915850 16084 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:17.916009 16084 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:17.916610 16084 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:17.917091 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:04:17.917261 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:04:17.917829 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.917909 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:17.918011 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:17.918255 16084 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:17.918709 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:17.934672 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:17.962471 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:17.972322 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:17.981015 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.073864 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.081537 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:18.085677 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.175263 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.261237 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.299937 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:18.300431 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.315650 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.339536 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:18.347960 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:18.352375 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.352917 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.398948 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.433003 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.451047 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.467753 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.469890 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:18.484647 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.486523 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.516342 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:18.546283 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.561995 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.586544 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.694811 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.738889 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.778544 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:04:18.779338 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:18.803953 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.820902 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:18.826479 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.866616 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.876865 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:18.911724 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:18.950953 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:18.960510 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:18.981396 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:18.982528 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.055610 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:19.196352 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:19.256736 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.268630 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.309861 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.313422 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.332526 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.353852 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:19.358515 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.364691 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:19.364724 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.396593 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:19.448802 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.460757 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.500074 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.548266 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.569631 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:19.582960 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:19.655395 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:04:19.712178 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.746946 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.834156 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:19.893196 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:19.896086 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.910590 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:19.916045 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:19.936719 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:19.972549 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:19.980736 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:19.984885 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.096287 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:20.148943 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.231900 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.261658 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.360085 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.384301 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:20.404781 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:20.411769 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:20.414808 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:20.417940 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.426231 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.441202 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:20.511667 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:20.586294 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:20.638427 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.661741 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.697974 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.731807 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.744079 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.775084 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:20.782351 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.791080 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:20.795302 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.803732 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.811107 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.824790 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:20.827893 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:20.880812 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:20.937436 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:20.958560 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.019517 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:21.041884 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:21.064277 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:21.114041 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.117451 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:21.124182 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.198690 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.199180 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.248734 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.258486 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.265722 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:21.314411 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:21.364733 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.394184 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:21.400965 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:21.435463 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:04:21.461222 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.526744 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:21.583351 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:21.629487 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.653120 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:21.682792 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.702862 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:21.716979 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.766117 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:21.783321 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:21.815582 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.817490 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.837081 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:21.853199 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:21.854924 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:21.912865 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:21.915239 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:21.992988 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:22.059435 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.064484 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.108680 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:22.205529 7567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:04:22.213312 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:22.280122 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:22.294756 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:22.311331 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:22.359079 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:22.366681 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:22.383260 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:22.411965 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:04:22.512614 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:22.549538 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:22.595436 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:04:22.595660 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.600103 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:22.625548 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:22.699127 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.758512 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:22.767766 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:22.799258 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.832675 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:22.886518 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:22.891809 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:22.904214 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:22.905678 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:22.961282 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.031229 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.068236 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.126358 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.202425 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:23.231014 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:23.233336 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.239492 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.246095 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:23.260480 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.276899 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.297626 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:23.378010 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:23.380231 17566 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:04:23.380362 17566 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:23.380925 17566 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:23.381338 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:04:23.381511 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:04:23.382025 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:23.382333 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:23.382401 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:23.382519 17566 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:23.407750 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.448824 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.482777 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.510687 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.542213 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.600231 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.628825 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.660749 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.667467 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:23.675073 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.693203 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.713800 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.735821 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.784268 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:23.804823 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:23.805294 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.826433 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:23.838265 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:04:23.862723 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.868571 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.885452 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:23.897372 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:23.986459 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:23.996244 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:24.071822 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:24.136755 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:24.194067 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.194756 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:24.251045 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.261528 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:24.275691 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:24.287720 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:24.292724 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.315743 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:24.324206 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:24.398885 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:24.418462 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:24.448801 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:24.488600 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.520687 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.532994 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:24.605098 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:24.644968 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:24.667438 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:24.734794 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:24.777580 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:04:24.784533 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:24.809623 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:24.812170 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:25.000247 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.018710 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:25.023123 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.023146 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:25.123018 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.123149 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.177719 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.209689 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.218327 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:25.220716 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.239195 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:25.251339 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.347105 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.357925 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.390568 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.399514 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:25.427414 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:25.430789 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.508962 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.525076 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.542609 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.615368 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:25.617018 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:25.623373 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.700491 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.734719 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:25.741127 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.778183 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:25.839916 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:25.871400 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:25.938647 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:25.953064 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:26.006086 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.011734 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:26.032950 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:26.033600 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.050016 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.081425 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.130007 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.160044 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.201968 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:26.288543 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.318084 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.323483 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.330055 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.330947 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:26.348266 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.377979 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:26.435508 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.449545 7177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:04:26.450134 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:26.475522 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.500416 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.532102 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.538609 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:26.593653 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.605544 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.618077 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:26.637781 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.669037 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:26.716810 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:26.808266 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:26.820691 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:26.844525 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:26.847189 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.863919 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.884904 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:26.942308 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.006826 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:27.043800 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.105079 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.110896 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:27.123728 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:27.136737 17627 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:27.136914 17627 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:27.137406 17627 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:27.137948 7237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:04:27.138121 7237 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:04:27.138553 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:27.138995 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:27.139051 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:27.139173 17627 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:27.176052 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.186726 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.253135 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:04:27.261158 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:27.279129 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.385910 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:27.395319 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.446054 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.474715 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:27.513974 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.553545 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.555590 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:04:27.585469 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:27.587461 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:27.600824 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:04:27.611191 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.636492 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.658064 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:27.711120 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.737181 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.776479 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:27.854425 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.908835 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:27.917716 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.972323 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:27.993669 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:28.006304 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:28.006470 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:28.081733 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:28.101706 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.113101 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:28.208745 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:28.224622 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.225023 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.232002 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:28.278694 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.284029 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:28.306147 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:28.359918 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.364982 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.446758 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:28.506855 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.536084 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:28.538937 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:28.579738 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.602298 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:28.612406 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:28.613464 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:28.635601 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:28.657474 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:28.706655 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:28.711337 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.792891 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:28.848246 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:04:28.879770 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:28.884356 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:28.898788 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:28.955053 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:28.967691 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.052893 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:29.103497 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:29.113826 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:29.121377 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.134532 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.153075 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.172487 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:29.192159 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:29.192409 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:29.195134 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:29.223778 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.232457 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.234645 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.263602 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.321900 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:29.328883 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.350000 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:29.596225 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.647900 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:29.660184 18495 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:04:29.660338 18495 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:29.660894 18495 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:29.661285 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:04:29.661420 7633 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:04:29.662077 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.662278 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:29.662330 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:29.662441 18495 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:29.669997 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.722595 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:29.726845 18401 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:29.726982 18401 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:29.727488 18401 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
W20240503 02:04:29.728138 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:29.728248 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:04:29.728399 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:04:29.728528 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.728956 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:29.729022 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:29.729137 18401 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:29.742882 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.752707 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:29.829530 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:29.872256 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:29.877084 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:29.898952 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:29.920185 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.102851 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.127720 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.145035 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.146286 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:30.154531 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.169180 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:30.213701 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.239336 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.273005 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.282171 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.293872 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:04:30.302770 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:30.363085 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.373395 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:30.381455 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.392807 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.418798 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.593564 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:30.619722 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.625783 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.668077 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.668403 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:30.689844 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.750981 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.752570 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:30.790247 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:30.817767 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.881279 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:30.897338 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:30.901213 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:30.956317 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:30.980394 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:30.983722 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.000695 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.007751 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.072032 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.099743 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:31.126013 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.138875 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.148743 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.151592 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:31.181319 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.186465 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.196851 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:31.237036 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.285738 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:31.331454 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.340232 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.367607 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:31.386741 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.489704 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.564739 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:04:31.573895 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:31.586390 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.594965 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.601233 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.646519 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:31.668607 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.672907 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.711956 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:31.718104 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:31.725107 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:31.732863 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:31.769838 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:31.771023 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:31.777051 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:31.966944 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.007905 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.025018 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.086308 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:32.163673 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:32.179867 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:32.183318 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:32.192339 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.198619 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.260161 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240503 02:04:32.260793 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:32.262430 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:32.381958 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.440843 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.474673 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.588923 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:32.614007 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:04:32.653220 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.660339 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:32.671218 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.674544 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:04:32.688882 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:32.695010 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:32.696875 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.705473 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:32.720644 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:32.825327 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:32.869367 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:32.870293 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:32.929248 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:32.934530 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.041203 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.060590 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.100344 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.113487 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.161845 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:33.182092 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:33.257475 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.269724 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.272015 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.299985 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:33.335925 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.415279 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.438783 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:33.468336 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.478750 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.500710 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.504439 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.538424 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:33.550340 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.563678 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.581831 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.620669 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.623309 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:33.688907 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:33.695493 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.714147 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.728785 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.739563 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:33.749984 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.797911 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:33.797974 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.835731 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:33.836257 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.853296 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 02:04:33.853940 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:33.905368 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:33.935209 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:33.961051 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:33.991149 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:34.070669 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:34.088193 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.107158 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:34.137094 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:34.220548 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:34.224855 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:34.271809 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:34.321177 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:34.340114 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:34.351419 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:34.355019 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.369228 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.453707 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:34.457813 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:34.537617 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:34.547152 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:34.588464 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:34.641989 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.699676 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.789505 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:34.818504 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:34.847726 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.852926 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:34.880035 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:34.900380 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:34.903894 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:34.957653 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.009352 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.019896 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:35.026441 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.102166 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:35.153858 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.187417 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:35.273510 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:04:35.275163 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.279349 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:35.339807 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.385196 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.400732 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.401903 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.445495 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:35.447844 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.482131 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.513684 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:35.534315 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.580931 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.621630 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:35.634343 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.666975 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:35.687633 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.731010 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.772554 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.786836 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:35.800909 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.810175 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.877147 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:35.893215 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:35.908252 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:35.933693 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.043301 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.061862 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:36.067727 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.069730 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:36.103824 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.117000 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:36.138141 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.158787 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.210204 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:36.235450 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.249073 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:36.282086 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.288856 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.314693 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:36.318367 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.323155 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.543180 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.554505 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:36.578533 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:04:36.579087 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:36.588105 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.601652 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:36.618952 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:36.625154 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.666411 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:36.667589 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.667608 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.713194 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.721872 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.758303 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.825996 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:36.877357 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:36.977121 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:36.977874 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:36.986018 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.003504 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.098917 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:37.132267 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.136332 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:37.139349 18401 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:37.139457 18401 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:37.139981 18401 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:37.140395 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:04:37.140552 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:04:37.141057 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.141350 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:37.141440 7174 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:37.141572 18401 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:37.168557 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.215535 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:37.240634 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.255568 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.270092 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 192 similar messages]
W20240503 02:04:37.292044 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.365916 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:37.370914 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.420632 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.420776 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.470629 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:37.470624 19477 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:37.470748 19477 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:37.471331 19477 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:37.471611 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:04:37.471772 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:04:37.472308 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:37.472379 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:04:37.472482 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:37.472555 19477 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:37.512095 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:37.589460 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:04:37.640126 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:37.644752 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:04:37.677227 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.694259 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:37.701789 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.729324 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.753655 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.753703 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.806162 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.835027 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:37.882025 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:37.933343 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:37.982107 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.003491 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:38.019795 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:38.056280 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:38.124490 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.144860 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:38.181142 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.244598 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:38.268569 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.274317 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:38.289651 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.301409 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:38.315071 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.354290 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.356364 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.358726 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.418844 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.448865 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:38.460965 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.565995 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:38.566006 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:38.567878 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.594727 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:38.625026 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:38.659886 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:38.679711 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.733490 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:38.791635 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:38.808737 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:38.834003 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.846441 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.907167 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:04:38.907918 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:38.970530 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:38.994587 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.020237 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:39.020723 19477 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:39.020845 19477 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:39.021325 19477 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:39.021646 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:04:39.021857 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:04:39.022315 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:39.022390 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:39.022528 19477 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:39.022812 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:39.023772 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:39.073812 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.098214 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:39.142464 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.155870 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:39.174211 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:39.204924 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.225687 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.233304 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:39.237733 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.253250 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:39.352263 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.357215 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:39.363673 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.372179 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:39.406941 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:39.547804 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.548733 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:39.552726 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:39.604475 7570 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.786918 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.789135 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:39.791229 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:39.806077 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:39.806739 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:39.900700 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:39.931524 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:40.009685 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.024480 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.093569 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:40.116518 19728 consensus_queue.cc:579] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.334s)
W20240503 02:04:40.117709 7570 consensus_peers.cc:462] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:40.120404 18401 consensus_queue.cc:579] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.214s)
I20240503 02:04:40.122308 7633 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } }
I20240503 02:04:40.123739 7237 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:04:40.124112 7502 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:04:40.124131 19883 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:04:40.124490 19945 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:04:40.126597 19883 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } }
I20240503 02:04:40.127070 7237 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } }
I20240503 02:04:40.128408 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.129102 7237 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:40.129207 7502 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } }
I20240503 02:04:40.131007 7371 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:40.131145 7098 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 4 to 5, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } }
W20240503 02:04:40.131287 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.131484 18401 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:04:40.132616 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.133409 19948 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.133821 7371 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.134651 7085 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a0a526f7fced4a04b4e0327f4656cf69 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.136134 7099 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.137645 7633 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
W20240503 02:04:40.138803 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240503 02:04:40.138864 7237 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:04:40.139364 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:40.139379 19728 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:04:40.139645 7502 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:04:40.140070 19728 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:04:40.140203 7570 consensus_peers.cc:485] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): Couldn't send request to peer bed387c0ff9241688cdf45f364dc6f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ba04b45ae1343f3aa4e5a82bbf7f486. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.141049 19945 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:40.141422 7237 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
W20240503 02:04:40.141888 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:40.142436 7502 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:40.143721 7084 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.144094 7094 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 5 to 6, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
W20240503 02:04:40.154970 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.182624 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.218966 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:04:40.235369 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:40.245884 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:40.277586 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:40.331898 19728 consensus_queue.cc:579] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.246s)
W20240503 02:04:40.376711 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:40.377249 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:04:40.382850 7306 consensus_peers.cc:462] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:40.392416 7568 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:40.432953 19821 consensus_queue.cc:579] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.014s)
W20240503 02:04:40.434649 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:40.439222 7502 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:40.440846 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 02:04:40.441231 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): Couldn't send request to peer bed387c0ff9241688cdf45f364dc6f5a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.441279 20021 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:04:40.441500 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.443094 20019 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:40.443497 7633 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:40.444643 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.446105 7091 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 9 to 10, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
W20240503 02:04:40.461182 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.500564 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.522075 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:40.527772 20023 ts_tablet_manager.cc:902] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 02:04:40.528286 20023 tablet_copy_client.cc:308] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 02:04:40.528777 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet a0a526f7fced4a04b4e0327f4656cf69 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:46763)
I20240503 02:04:40.528884 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet a0a526f7fced4a04b4e0327f4656cf69 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:46763: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-a0a526f7fced4a04b4e0327f4656cf69
I20240503 02:04:40.530100 7261 tablet_copy_source_session.cc:205] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:40.531054 20023 ts_tablet_manager.cc:1875] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:40.533082 20023 ts_tablet_manager.cc:1888] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:04:40.533255 20023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0a526f7fced4a04b4e0327f4656cf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:40.534189 20023 tablet_copy_client.cc:783] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:40.538965 20023 tablet_copy_client.cc:647] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:40.540252 20023 tablet_copy_client.cc:523] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:40.541110 19728 consensus_queue.cc:579] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.165s)
I20240503 02:04:40.541949 20023 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 02:04:40.544291 7633 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:40.545771 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:40.546308 19728 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:04:40.546551 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.548172 19883 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 02:04:40.549557 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec179e5e683d4ba099c459331ea5e578 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.549670 7371 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } }
I20240503 02:04:40.550704 7099 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:40.552181 20023 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:40.552417 20023 tablet_bootstrap.cc:492] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 02:04:40.552671 20023 ts_tablet_manager.cc:1367] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:04:40.553184 20023 raft_consensus.cc:348] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:40.553407 20023 raft_consensus.cc:729] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 02:04:40.553562 20023 consensus_queue.cc:260] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:40.554477 20023 ts_tablet_manager.cc:1398] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:04:40.554405 7633 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:40.555763 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-a0a526f7fced4a04b4e0327f4656cf69 received from {username='slave'} at 127.0.99.131:46763
I20240503 02:04:40.555842 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-a0a526f7fced4a04b4e0327f4656cf69 on tablet a0a526f7fced4a04b4e0327f4656cf69 with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 02:04:40.555914 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:04:40.555958 7570 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec179e5e683d4ba099c459331ea5e578. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.556447 19728 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:04:40.557168 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.558338 19883 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.558775 7371 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.559739 7084 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ec179e5e683d4ba099c459331ea5e578 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.560596 7091 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 8 to 9, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:04:40.575991 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:40.651237 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69): 4921 bytes on disk
W20240503 02:04:40.651885 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:40.651870 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(a0a526f7fced4a04b4e0327f4656cf69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:04:40.668498 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:40.675009 20056 ts_tablet_manager.cc:902] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 02:04:40.675493 20056 tablet_copy_client.cc:308] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 02:04:40.676023 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 from peer bed387c0ff9241688cdf45f364dc6f5a ({username='slave'} at 127.0.99.130:59195)
I20240503 02:04:40.676138 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 from peer bed387c0ff9241688cdf45f364dc6f5a at {username='slave'} at 127.0.99.130:59195: session id = bed387c0ff9241688cdf45f364dc6f5a-5ba04b45ae1343f3aa4e5a82bbf7f486
I20240503 02:04:40.677403 7653 tablet_copy_source_session.cc:205] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:40.678519 20056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:40.680478 20056 tablet_copy_client.cc:783] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:40.685071 20056 tablet_copy_client.cc:647] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:40.686372 20056 tablet_copy_client.cc:523] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:40.688002 20056 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 02:04:40.695255 20056 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:40.695623 20056 tablet_bootstrap.cc:492] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap complete.
I20240503 02:04:40.695817 20056 ts_tablet_manager.cc:1367] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 02:04:40.696203 20056 raft_consensus.cc:348] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:40.696314 20056 raft_consensus.cc:729] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: LEARNER
I20240503 02:04:40.696442 20056 consensus_queue.cc:260] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:40.696714 20056 ts_tablet_manager.cc:1398] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:04:40.697510 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session bed387c0ff9241688cdf45f364dc6f5a-5ba04b45ae1343f3aa4e5a82bbf7f486 received from {username='slave'} at 127.0.99.130:59195
I20240503 02:04:40.697584 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session bed387c0ff9241688cdf45f364dc6f5a-5ba04b45ae1343f3aa4e5a82bbf7f486 on tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 with peer bed387c0ff9241688cdf45f364dc6f5a
W20240503 02:04:40.710115 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:40.726583 20011 consensus_queue.cc:579] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.284s)
I20240503 02:04:40.748740 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486): 4583 bytes on disk
I20240503 02:04:40.749258 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(5ba04b45ae1343f3aa4e5a82bbf7f486) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:04:40.753492 20012 consensus_queue.cc:579] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.041s)
I20240503 02:04:40.753953 20021 consensus_queue.cc:579] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.349s)
I20240503 02:04:40.759970 7371 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 02:04:40.761566 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:40.761968 20011 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:04:40.762383 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.762434 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.763517 7569 consensus_peers.cc:462] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:40.763844 20012 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:40.764294 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:40.765410 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.766801 7099 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 11 to 12, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:40.770197 7371 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
W20240503 02:04:40.771724 7308 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71f12a25a1db4650a6c37b5000a668c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.772131 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 02:04:40.772229 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.772521 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.772709 19822 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:40.774067 20012 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.774461 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.775424 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 71f12a25a1db4650a6c37b5000a668c7 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.776729 7099 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
W20240503 02:04:40.786347 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:40.843731 20056 ts_tablet_manager.cc:902] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:40.844285 20056 tablet_copy_client.cc:308] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:40.844846 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer bed387c0ff9241688cdf45f364dc6f5a ({username='slave'} at 127.0.99.130:46341)
I20240503 02:04:40.844970 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 from peer bed387c0ff9241688cdf45f364dc6f5a at {username='slave'} at 127.0.99.130:46341: session id = bed387c0ff9241688cdf45f364dc6f5a-27c539b143674bdd8e3e5aaa4b78c1e9
I20240503 02:04:40.846127 7523 tablet_copy_source_session.cc:205] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
W20240503 02:04:40.847419 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:40.847781 20056 ts_tablet_manager.cc:1875] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:40.849711 20056 ts_tablet_manager.cc:1888] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 02:04:40.849887 20056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c539b143674bdd8e3e5aaa4b78c1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:40.850812 20056 tablet_copy_client.cc:783] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:40.856170 20056 tablet_copy_client.cc:647] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:40.857578 20056 tablet_copy_client.cc:523] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:40.859135 20056 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 02:04:40.864099 19945 consensus_queue.cc:579] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.016s)
I20240503 02:04:40.867179 7633 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 02:04:40.869055 19945 consensus_queue.cc:579] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.427s)
I20240503 02:04:40.869120 7237 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 02:04:40.869530 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.870489 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.870587 19883 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:40.871886 20056 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 02:04:40.872109 7570 consensus_peers.cc:462] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:40.872311 20056 tablet_bootstrap.cc:492] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap complete.
I20240503 02:04:40.872339 7633 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:40.872575 20056 ts_tablet_manager.cc:1367] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.001s
I20240503 02:04:40.873977 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:40.874372 19728 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:04:40.874867 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.874799 19945 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:40.875203 20056 raft_consensus.cc:348] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:40.875398 20056 raft_consensus.cc:729] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: LEARNER
I20240503 02:04:40.875339 7237 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:40.875599 20056 consensus_queue.cc:260] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:40.875996 20056 ts_tablet_manager.cc:1398] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:04:40.876523 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81086ff737c84d9d9f04743d31c6151b with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.876691 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session bed387c0ff9241688cdf45f364dc6f5a-27c539b143674bdd8e3e5aaa4b78c1e9 received from {username='slave'} at 127.0.99.130:46341
I20240503 02:04:40.876760 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session bed387c0ff9241688cdf45f364dc6f5a-27c539b143674bdd8e3e5aaa4b78c1e9 on tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with peer bed387c0ff9241688cdf45f364dc6f5a
I20240503 02:04:40.878145 7091 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 10 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:40.879302 19945 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 02:04:40.880863 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0539dded7554723b9e918e8cdfbab20 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.880878 7502 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 02:04:40.883513 7091 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, NON_VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 02:04:40.884637 7633 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
W20240503 02:04:40.887451 7570 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921): Couldn't send request to peer 5b46f0d4b665497b8d7e57bf4d5f36cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.887935 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.888089 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.888108 7237 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:04:40.888468 19945 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:40.888579 20065 consensus_queue.cc:579] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.089s)
I20240503 02:04:40.888650 7633 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:40.889704 20021 consensus_queue.cc:579] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.048s)
W20240503 02:04:40.891191 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:40.891292 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0539dded7554723b9e918e8cdfbab20. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.891623 7502 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 02:04:40.891716 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.892009 7371 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 02:04:40.892081 20121 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:40.893023 19883 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.894037 19945 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:40.894191 7237 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:04:40.894412 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.894619 19822 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:40.895325 7084 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 81086ff737c84d9d9f04743d31c6151b with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.895535 7084 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f0539dded7554723b9e918e8cdfbab20 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.895857 7502 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:40.896183 7239 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } } }
I20240503 02:04:40.896761 7091 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 12 to 13, NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:40.897707 7500 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:40.898864 7099 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 11 to 12, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
W20240503 02:04:40.900144 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:40.900619 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.901646 7633 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:40.902043 20021 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:40.903285 19822 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
W20240503 02:04:40.904733 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.905112 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d00b8fcb29f47298643903ec0cba606 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:40.905371 7237 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 02:04:40.906368 7099 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 8 to 9, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:40.908005 20019 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:40.909905 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 527af716906b42729952db28e790bafd with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.910385 7371 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:40.910952 7099 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:04:40.912068 7306 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d00b8fcb29f47298643903ec0cba606. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.912376 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:40.912632 7633 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:40.912874 20065 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:04:40.913466 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:40.915378 20011 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:40.915881 7239 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:40.916900 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d00b8fcb29f47298643903ec0cba606 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:40.918422 7098 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
W20240503 02:04:40.930322 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:40.963016 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:41.001062 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9): 4581 bytes on disk
I20240503 02:04:41.001617 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(27c539b143674bdd8e3e5aaa4b78c1e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.028223 7500 raft_consensus.cc:1203] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:04:41.090613 20065 consensus_queue.cc:579] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.479s)
I20240503 02:04:41.107844 20123 consensus_queue.cc:579] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.252s)
I20240503 02:04:41.113037 20156 ts_tablet_manager.cc:902] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 02:04:41.113548 20156 tablet_copy_client.cc:308] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 02:04:41.114017 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet ec179e5e683d4ba099c459331ea5e578 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:49207)
I20240503 02:04:41.114117 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet ec179e5e683d4ba099c459331ea5e578 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:49207: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-ec179e5e683d4ba099c459331ea5e578
I20240503 02:04:41.115005 7653 tablet_copy_source_session.cc:205] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.115919 20156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec179e5e683d4ba099c459331ea5e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.118044 20156 tablet_copy_client.cc:783] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.122259 20156 tablet_copy_client.cc:647] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.123286 20156 tablet_copy_client.cc:523] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.124519 20156 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 02:04:41.132248 20156 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.132517 20156 tablet_bootstrap.cc:492] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 02:04:41.132673 20156 ts_tablet_manager.cc:1367] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 02:04:41.133025 20156 raft_consensus.cc:348] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.133134 20156 raft_consensus.cc:729] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 02:04:41.133229 20156 consensus_queue.cc:260] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.133513 20156 ts_tablet_manager.cc:1398] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:04:41.133978 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-ec179e5e683d4ba099c459331ea5e578 received from {username='slave'} at 127.0.99.131:49207
I20240503 02:04:41.134069 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-ec179e5e683d4ba099c459331ea5e578 on tablet ec179e5e683d4ba099c459331ea5e578 with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 02:04:41.135749 20065 consensus_queue.cc:579] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.432s)
I20240503 02:04:41.138218 7371 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
W20240503 02:04:41.139798 7307 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0b1a4b2ede6493f9e4e094649b4b57d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.139878 7633 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:41.140257 20012 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:04:41.140628 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.142071 20065 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:41.142473 7633 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:41.143587 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a0b1a4b2ede6493f9e4e094649b4b57d with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.144660 7099 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.154367 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578): 4838 bytes on disk
I20240503 02:04:41.154839 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(ec179e5e683d4ba099c459331ea5e578) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 02:04:41.159919 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 71f12a25a1db4650a6c37b5000a668c7: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:41.160467 20123 consensus_queue.cc:579] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.019s)
I20240503 02:04:41.163129 7633 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } }
I20240503 02:04:41.164764 7371 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 02:04:41.164990 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.165254 20124 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:41.167373 20125 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 02:04:41.167836 7371 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 02:04:41.169207 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68791b3adccd4168b55a2da4423232d7 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.170308 7098 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 10 to 12, NON_VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } }
I20240503 02:04:41.173599 7633 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
W20240503 02:04:41.175155 7569 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): Couldn't send request to peer 4356d1cb1050473a8d9e50ff09a24dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68791b3adccd4168b55a2da4423232d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.175150 7371 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
W20240503 02:04:41.175671 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.175880 20123 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:41.177330 19945 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:41.178799 7084 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 68791b3adccd4168b55a2da4423232d7 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.179431 7371 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } } }
I20240503 02:04:41.180027 7099 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 12 to 13, NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:04:41.181998 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:04:41.240918 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:04:41.279873 7371 raft_consensus.cc:1203] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240503 02:04:41.280414 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:41.282785 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.313622 20156 ts_tablet_manager.cc:902] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 02:04:41.314167 20156 tablet_copy_client.cc:308] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 02:04:41.314627 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet 81086ff737c84d9d9f04743d31c6151b from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:49207)
I20240503 02:04:41.314729 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet 81086ff737c84d9d9f04743d31c6151b from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:49207: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-81086ff737c84d9d9f04743d31c6151b
I20240503 02:04:41.315896 7653 tablet_copy_source_session.cc:205] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.316923 20156 ts_tablet_manager.cc:1875] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 02:04:41.318843 7569 consensus_peers.cc:462] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:41.319063 20156 ts_tablet_manager.cc:1888] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 02:04:41.319255 20156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81086ff737c84d9d9f04743d31c6151b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.320340 20156 tablet_copy_client.cc:783] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.324700 20156 tablet_copy_client.cc:647] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.326079 20156 tablet_copy_client.cc:523] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.327636 20156 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 02:04:41.335803 20124 consensus_queue.cc:579] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.479s)
I20240503 02:04:41.337764 20191 ts_tablet_manager.cc:902] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 02:04:41.338405 20191 tablet_copy_client.cc:308] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 02:04:41.338529 20156 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.338801 20156 tablet_bootstrap.cc:492] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 02:04:41.338868 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 71f12a25a1db4650a6c37b5000a668c7 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb ({username='slave'} at 127.0.99.131:51591)
I20240503 02:04:41.338963 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 71f12a25a1db4650a6c37b5000a668c7 from peer 5b46f0d4b665497b8d7e57bf4d5f36cb at {username='slave'} at 127.0.99.131:51591: session id = 5b46f0d4b665497b8d7e57bf4d5f36cb-71f12a25a1db4650a6c37b5000a668c7
I20240503 02:04:41.338991 20156 ts_tablet_manager.cc:1367] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240503 02:04:41.339339 20156 raft_consensus.cc:348] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.339450 20156 raft_consensus.cc:729] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 02:04:41.339534 20156 consensus_queue.cc:260] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.339833 20156 ts_tablet_manager.cc:1398] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 02:04:41.340090 7392 tablet_copy_source_session.cc:205] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.340601 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-81086ff737c84d9d9f04743d31c6151b received from {username='slave'} at 127.0.99.131:49207
I20240503 02:04:41.340673 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-81086ff737c84d9d9f04743d31c6151b on tablet 81086ff737c84d9d9f04743d31c6151b with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 02:04:41.340999 20191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f12a25a1db4650a6c37b5000a668c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.342965 20191 tablet_copy_client.cc:783] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.347292 20191 tablet_copy_client.cc:647] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.348551 20191 tablet_copy_client.cc:523] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.349733 20191 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap starting.
I20240503 02:04:41.358775 20191 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.359086 20191 tablet_bootstrap.cc:492] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Bootstrap complete.
I20240503 02:04:41.359272 20191 ts_tablet_manager.cc:1367] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 02:04:41.359755 20191 raft_consensus.cc:348] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.359869 20191 raft_consensus.cc:729] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b46f0d4b665497b8d7e57bf4d5f36cb, State: Initialized, Role: LEARNER
I20240503 02:04:41.359959 20191 consensus_queue.cc:260] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: true } }
I20240503 02:04:41.360248 20191 ts_tablet_manager.cc:1398] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:04:41.361088 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-71f12a25a1db4650a6c37b5000a668c7 received from {username='slave'} at 127.0.99.131:51591
I20240503 02:04:41.361182 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 5b46f0d4b665497b8d7e57bf4d5f36cb-71f12a25a1db4650a6c37b5000a668c7 on tablet 71f12a25a1db4650a6c37b5000a668c7 with peer 5b46f0d4b665497b8d7e57bf4d5f36cb
I20240503 02:04:41.361335 7371 raft_consensus.cc:1203] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 02:04:41.361898 19945 consensus_queue.cc:579] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.153s)
W20240503 02:04:41.380524 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.393388 20124 consensus_queue.cc:579] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.102s)
W20240503 02:04:41.394708 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:41.406363 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b): 4923 bytes on disk
I20240503 02:04:41.406986 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(81086ff737c84d9d9f04743d31c6151b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.407832 7550 maintenance_manager.cc:419] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Scheduling UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7): 4583 bytes on disk
I20240503 02:04:41.408169 7440 maintenance_manager.cc:643] P 5b46f0d4b665497b8d7e57bf4d5f36cb: UndoDeltaBlockGCOp(71f12a25a1db4650a6c37b5000a668c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.479990 20021 consensus_queue.cc:579] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.151s)
W20240503 02:04:41.481249 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:41.482560 7500 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 02:04:41.484004 7239 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:04:41.484258 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.484406 20021 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:41.486093 20132 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 02:04:41.486481 7239 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 02:04:41.487627 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.488894 7098 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:04:41.491688 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.493000 7500 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:41.494238 7239 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 02:04:41.494529 7437 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:41.494774 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.494870 20132 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:41.496282 20133 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.496698 7239 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.497807 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.498821 7098 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 9 to 10, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.505591 20200 ts_tablet_manager.cc:902] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 02:04:41.506078 20200 tablet_copy_client.cc:308] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 02:04:41.509270 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet f0539dded7554723b9e918e8cdfbab20 from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:42965)
I20240503 02:04:41.509366 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet f0539dded7554723b9e918e8cdfbab20 from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:42965: session id = 4356d1cb1050473a8d9e50ff09a24dc2-f0539dded7554723b9e918e8cdfbab20
I20240503 02:04:41.510202 7653 tablet_copy_source_session.cc:205] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.511165 20200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0539dded7554723b9e918e8cdfbab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.512830 20205 ts_tablet_manager.cc:902] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:41.513350 20200 tablet_copy_client.cc:783] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.513422 20205 tablet_copy_client.cc:308] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:41.513931 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 527af716906b42729952db28e790bafd from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:51345)
I20240503 02:04:41.514030 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 527af716906b42729952db28e790bafd from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:51345: session id = 4356d1cb1050473a8d9e50ff09a24dc2-527af716906b42729952db28e790bafd
I20240503 02:04:41.514942 7523 tablet_copy_source_session.cc:205] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.515627 20205 ts_tablet_manager.cc:1875] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:41.517700 20205 ts_tablet_manager.cc:1888] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:04:41.517874 20205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527af716906b42729952db28e790bafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.518214 20200 tablet_copy_client.cc:647] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.518837 20205 tablet_copy_client.cc:783] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.520367 20200 tablet_copy_client.cc:523] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.521831 20210 ts_tablet_manager.cc:902] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 02:04:41.521908 20200 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 02:04:41.522426 20210 tablet_copy_client.cc:308] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 02:04:41.523025 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 0d00b8fcb29f47298643903ec0cba606 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:51673)
I20240503 02:04:41.523128 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 0d00b8fcb29f47298643903ec0cba606 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:51673: session id = efeb87c28ee04b3da5c686665b0c5c58-0d00b8fcb29f47298643903ec0cba606
I20240503 02:04:41.524230 20205 tablet_copy_client.cc:647] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.524276 7392 tablet_copy_source_session.cc:205] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.525395 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d00b8fcb29f47298643903ec0cba606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.525740 20215 ts_tablet_manager.cc:902] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 02:04:41.526087 20205 tablet_copy_client.cc:523] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.526310 20215 tablet_copy_client.cc:308] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 02:04:41.526757 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:50331)
I20240503 02:04:41.526844 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet a0b1a4b2ede6493f9e4e094649b4b57d from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:50331: session id = 4356d1cb1050473a8d9e50ff09a24dc2-a0b1a4b2ede6493f9e4e094649b4b57d
I20240503 02:04:41.527376 20205 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 02:04:41.527736 7392 tablet_copy_source_session.cc:205] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.528293 20210 tablet_copy_client.cc:783] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.528483 20215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b1a4b2ede6493f9e4e094649b4b57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.531726 20215 tablet_copy_client.cc:783] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.533938 20210 tablet_copy_client.cc:647] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.534399 20124 consensus_queue.cc:579] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.105s)
I20240503 02:04:41.536284 20210 tablet_copy_client.cc:523] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.537339 7633 consensus_queue.cc:237] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } }
I20240503 02:04:41.538039 20210 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:41.539191 7239 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:41.539681 20124 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:04:41.540953 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.540966 20205 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.541179 20200 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.541213 20205 tablet_bootstrap.cc:492] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 02:04:41.541388 20205 ts_tablet_manager.cc:1367] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240503 02:04:41.541415 20200 tablet_bootstrap.cc:492] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 02:04:41.541522 7500 raft_consensus.cc:1261] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:41.541587 20200 ts_tablet_manager.cc:1367] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20240503 02:04:41.541822 20205 raft_consensus.cc:348] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.541945 20205 raft_consensus.cc:729] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 02:04:41.541929 20124 consensus_queue.cc:1035] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:41.542043 20200 raft_consensus.cc:348] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.542047 20205 consensus_queue.cc:260] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.542151 20200 raft_consensus.cc:729] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 02:04:41.542246 20200 consensus_queue.cc:260] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.542382 20205 ts_tablet_manager.cc:1398] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:04:41.542701 20200 ts_tablet_manager.cc:1398] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:04:41.542922 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-527af716906b42729952db28e790bafd received from {username='slave'} at 127.0.99.129:51345
I20240503 02:04:41.542989 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-527af716906b42729952db28e790bafd on tablet 527af716906b42729952db28e790bafd with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 02:04:41.544556 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-f0539dded7554723b9e918e8cdfbab20 received from {username='slave'} at 127.0.99.129:42965
I20240503 02:04:41.544629 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-f0539dded7554723b9e918e8cdfbab20 on tablet f0539dded7554723b9e918e8cdfbab20 with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 02:04:41.546954 19945 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:41.547353 7239 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:41.548770 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4628afbfeff4f8bbaba5da1e89e4850 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.549261 7500 raft_consensus.cc:2932] T b4628afbfeff4f8bbaba5da1e89e4850 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } }
I20240503 02:04:41.549633 7099 catalog_manager.cc:5484] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.555725 20215 tablet_copy_client.cc:647] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.557375 20215 tablet_copy_client.cc:523] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.557668 20210 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.558312 20210 tablet_bootstrap.cc:492] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:41.558530 20210 ts_tablet_manager.cc:1367] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.005s
I20240503 02:04:41.559116 20215 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 02:04:41.559232 20210 raft_consensus.cc:348] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:41.559355 20210 raft_consensus.cc:729] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:41.559459 20210 consensus_queue.cc:260] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:41.559777 20210 ts_tablet_manager.cc:1398] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:04:41.560549 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-0d00b8fcb29f47298643903ec0cba606 received from {username='slave'} at 127.0.99.132:51673
I20240503 02:04:41.560623 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-0d00b8fcb29f47298643903ec0cba606 on tablet 0d00b8fcb29f47298643903ec0cba606 with peer efeb87c28ee04b3da5c686665b0c5c58
W20240503 02:04:41.562505 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:41.562894 20135 raft_consensus.cc:1050] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 02:04:41.563324 20135 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 02:04:41.565097 7371 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:41.565498 7500 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:04:41.565981 7177 consensus_peers.cc:485] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.565994 20237 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:41.566172 20135 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:41.576242 20135 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.576747 7371 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.578439 7500 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.579385 7099 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 7 to 8, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.580065 20215 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.580384 20215 tablet_bootstrap.cc:492] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 02:04:41.580546 20215 ts_tablet_manager.cc:1367] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240503 02:04:41.581002 20215 raft_consensus.cc:348] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.581126 20215 raft_consensus.cc:729] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 02:04:41.581219 20215 consensus_queue.cc:260] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.581597 20215 ts_tablet_manager.cc:1398] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:04:41.582072 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-a0b1a4b2ede6493f9e4e094649b4b57d received from {username='slave'} at 127.0.99.129:50331
I20240503 02:04:41.582140 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-a0b1a4b2ede6493f9e4e094649b4b57d on tablet a0b1a4b2ede6493f9e4e094649b4b57d with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 02:04:41.584059 7239 consensus_queue.cc:237] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 02:04:41.585299 7371 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:41.585364 7500 raft_consensus.cc:1261] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:41.585682 20237 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:41.585811 20135 consensus_queue.cc:1035] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:41.587515 20135 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.588068 7500 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.588191 7371 raft_consensus.cc:2932] T a0a526f7fced4a04b4e0327f4656cf69 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:41.589073 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0a526f7fced4a04b4e0327f4656cf69 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.590624 7099 catalog_manager.cc:5484] T a0a526f7fced4a04b4e0327f4656cf69 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 8 to 9, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:41.616268 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d): 4923 bytes on disk
I20240503 02:04:41.616777 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(a0b1a4b2ede6493f9e4e094649b4b57d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.617594 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20): 4921 bytes on disk
I20240503 02:04:41.618029 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(f0539dded7554723b9e918e8cdfbab20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.618757 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd): 4839 bytes on disk
I20240503 02:04:41.619060 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(527af716906b42729952db28e790bafd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.632544 20215 ts_tablet_manager.cc:902] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Initiating tablet copy from peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123)
I20240503 02:04:41.633096 20215 tablet_copy_client.cc:308] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33123
I20240503 02:04:41.633626 7653 tablet_copy_service.cc:140] P efeb87c28ee04b3da5c686665b0c5c58: Received BeginTabletCopySession request for tablet 68791b3adccd4168b55a2da4423232d7 from peer 4356d1cb1050473a8d9e50ff09a24dc2 ({username='slave'} at 127.0.99.129:42965)
I20240503 02:04:41.633754 7653 tablet_copy_service.cc:161] P efeb87c28ee04b3da5c686665b0c5c58: Beginning new tablet copy session on tablet 68791b3adccd4168b55a2da4423232d7 from peer 4356d1cb1050473a8d9e50ff09a24dc2 at {username='slave'} at 127.0.99.129:42965: session id = 4356d1cb1050473a8d9e50ff09a24dc2-68791b3adccd4168b55a2da4423232d7
I20240503 02:04:41.634828 7653 tablet_copy_source_session.cc:205] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: Tablet Copy: opened 5 blocks and 1 log segments
W20240503 02:04:41.635632 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:41.635895 20215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68791b3adccd4168b55a2da4423232d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:41.637756 20215 tablet_copy_client.cc:783] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:41.642004 20215 tablet_copy_client.cc:647] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:41.643206 20215 tablet_copy_client.cc:523] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:41.644552 20215 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap starting.
I20240503 02:04:41.652630 20215 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:41.652913 20215 tablet_bootstrap.cc:492] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Bootstrap complete.
I20240503 02:04:41.653086 20215 ts_tablet_manager.cc:1367] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:04:41.653510 20215 raft_consensus.cc:348] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.653617 20215 raft_consensus.cc:729] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4356d1cb1050473a8d9e50ff09a24dc2, State: Initialized, Role: LEARNER
I20240503 02:04:41.653705 20215 consensus_queue.cc:260] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: true } }
I20240503 02:04:41.654001 20215 ts_tablet_manager.cc:1398] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:04:41.654714 7653 tablet_copy_service.cc:342] P efeb87c28ee04b3da5c686665b0c5c58: Request end of tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-68791b3adccd4168b55a2da4423232d7 received from {username='slave'} at 127.0.99.129:42965
I20240503 02:04:41.654814 7653 tablet_copy_service.cc:433] P efeb87c28ee04b3da5c686665b0c5c58: ending tablet copy session 4356d1cb1050473a8d9e50ff09a24dc2-68791b3adccd4168b55a2da4423232d7 on tablet 68791b3adccd4168b55a2da4423232d7 with peer 4356d1cb1050473a8d9e50ff09a24dc2
I20240503 02:04:41.655347 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606): 4921 bytes on disk
I20240503 02:04:41.656199 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(0d00b8fcb29f47298643903ec0cba606) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.662992 7500 raft_consensus.cc:1203] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:04:41.680946 20021 consensus_queue.cc:579] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.265s)
W20240503 02:04:41.682210 7438 consensus_peers.cc:462] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:41.683495 7500 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } }
I20240503 02:04:41.684990 7371 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:04:41.685025 7239 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:04:41.685417 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.685444 20021 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:04:41.685556 20133 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:04:41.687372 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.687594 20132 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 02:04:41.688025 7239 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 02:04:41.688220 7371 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } }
I20240503 02:04:41.689592 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 245cba4e5728491896e7995ca0d59b36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.690196 7099 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 5 to 6, NON_VOTER c778431b13784c7c81f3924b7f73e664 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.694013 7500 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:41.695533 7371 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:41.695921 20133 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:04:41.696393 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 245cba4e5728491896e7995ca0d59b36. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:41.697108 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.697654 7239 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:41.697997 20133 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:41.698241 20132 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.698734 7371 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.698971 7239 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.699831 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 245cba4e5728491896e7995ca0d59b36 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.701146 7099 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.702318 20237 consensus_queue.cc:579] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.303s)
W20240503 02:04:41.702955 7174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:04:41.703716 7174 consensus_peers.cc:462] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:04:41.706260 7239 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:41.707619 7500 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:04:41.708016 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.708446 20135 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:41.710333 20237 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 02:04:41.710794 7500 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } }
I20240503 02:04:41.712292 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66317a254eb644359f65610368d84d76 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.712908 7091 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 7 to 8, NON_VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.715979 7239 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
W20240503 02:04:41.717530 7175 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66317a254eb644359f65610368d84d76. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:04:41.718045 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.718930 7500 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:41.719329 20135 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:41.720994 20243 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.721455 7500 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.722676 7085 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 66317a254eb644359f65610368d84d76 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.724025 7091 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 8 to 9, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.757915 7500 raft_consensus.cc:1203] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240503 02:04:41.773849 20263 consensus_queue.cc:579] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.245s)
W20240503 02:04:41.775169 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.828527 20126 raft_consensus.cc:1050] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER bed387c0ff9241688cdf45f364dc6f5a to VOTER
I20240503 02:04:41.829031 20126 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:41.830351 7239 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:41.830526 7500 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:41.830767 7371 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:04:41.830770 20126 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:41.831070 20123 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:41.831207 20126 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:41.833195 20124 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.833676 7371 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.833678 7500 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.835430 7239 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.836328 7091 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 6 to 7, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.839010 20021 consensus_queue.cc:579] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.124s)
I20240503 02:04:41.841317 7633 consensus_queue.cc:237] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:41.842613 7500 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:41.842615 7371 raft_consensus.cc:1261] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:41.843086 20123 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:41.843307 20126 consensus_queue.cc:1035] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:41.844952 20263 consensus_queue.cc:579] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.129s)
I20240503 02:04:41.845180 20125 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.845959 7371 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.846967 7084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.845674 7500 raft_consensus.cc:2932] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
W20240503 02:04:41.847707 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:04:41.848769 7094 catalog_manager.cc:5484] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:41.851488 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35818
I20240503 02:04:41.852325 20288 tablet_replica.cc:330] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 02:04:41.852551 20288 raft_consensus.cc:2229] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:41.852782 20288 raft_consensus.cc:2258] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:41.853581 20288 ts_tablet_manager.cc:1875] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:04:41.856706 20288 ts_tablet_manager.cc:1888] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:04:41.856824 20288 log.cc:1196] T 5ba04b45ae1343f3aa4e5a82bbf7f486 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ba04b45ae1343f3aa4e5a82bbf7f486
I20240503 02:04:41.857475 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 5ba04b45ae1343f3aa4e5a82bbf7f486 (table rebalance_test_table_6 [id=135581cd6a7f43aeb219da3a660f73dc]) successfully deleted
W20240503 02:04:41.859973 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.870570 7288 maintenance_manager.cc:419] P 4356d1cb1050473a8d9e50ff09a24dc2: Scheduling UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7): 4923 bytes on disk
I20240503 02:04:41.871052 7178 maintenance_manager.cc:643] P 4356d1cb1050473a8d9e50ff09a24dc2: UndoDeltaBlockGCOp(68791b3adccd4168b55a2da4423232d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 02:04:41.897199 7500 raft_consensus.cc:1203] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.12->[2.13-2.13] Dedup: 2.13->[]
I20240503 02:04:41.913340 7239 raft_consensus.cc:1203] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240503 02:04:41.939635 7633 raft_consensus.cc:1203] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 02:04:41.941799 20132 consensus_queue.cc:579] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.056s)
W20240503 02:04:41.945111 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:41.945050 7500 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
W20240503 02:04:41.946703 7437 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.946918 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:04:41.947129 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.947377 20199 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:41.949318 20021 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.950847 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c18129344d124a5f8f688ec35ef69a55 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:41.951027 7239 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:41.952122 7094 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 6 to 7, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:41.976156 20199 raft_consensus.cc:1050] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER bed387c0ff9241688cdf45f364dc6f5a to VOTER
I20240503 02:04:41.976612 20199 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:41.978189 7371 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:41.978176 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:41.978591 20199 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:04:41.978721 20133 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:04:41.979030 7439 consensus_peers.cc:485] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:41.980986 20199 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.981411 7633 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.982983 7371 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.984406 7098 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:41.987432 7500 consensus_queue.cc:237] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:41.988611 7371 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:41.988718 7633 raft_consensus.cc:1261] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:41.989073 20133 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:41.989187 20021 consensus_queue.cc:1035] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:41.990588 20021 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.991036 7371 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.991372 7633 raft_consensus.cc:2932] T 27c539b143674bdd8e3e5aaa4b78c1e9 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.993535 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27c539b143674bdd8e3e5aaa4b78c1e9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:41.994516 20210 ts_tablet_manager.cc:902] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:41.995054 20210 tablet_copy_client.cc:308] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:41.994343 7098 catalog_manager.cc:5484] T 27c539b143674bdd8e3e5aaa4b78c1e9 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 11 to 12, VOTER c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:41.995478 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:34781)
I20240503 02:04:41.995591 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:34781: session id = efeb87c28ee04b3da5c686665b0c5c58-8ebe541161c84bd6b11bcd3d13d3bd2e
I20240503 02:04:41.996549 7523 tablet_copy_source_session.cc:205] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:41.997229 20210 ts_tablet_manager.cc:1875] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:41.999495 20210 ts_tablet_manager.cc:1888] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:04:41.999711 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebe541161c84bd6b11bcd3d13d3bd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.001379 20210 tablet_copy_client.cc:783] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:42.006961 20210 tablet_copy_client.cc:647] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.008410 20210 tablet_copy_client.cc:523] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.009855 20210 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:42.020988 20210 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.021302 20210 tablet_bootstrap.cc:492] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:42.021486 20210 ts_tablet_manager.cc:1367] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20240503 02:04:42.021939 20210 raft_consensus.cc:348] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.022078 20210 raft_consensus.cc:729] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:42.022200 20210 consensus_queue.cc:260] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.022612 20210 ts_tablet_manager.cc:1398] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:04:42.023053 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-8ebe541161c84bd6b11bcd3d13d3bd2e received from {username='slave'} at 127.0.99.132:34781
I20240503 02:04:42.023133 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-8ebe541161c84bd6b11bcd3d13d3bd2e on tablet 8ebe541161c84bd6b11bcd3d13d3bd2e with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:42.039927 19822 consensus_queue.cc:579] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.193s)
I20240503 02:04:42.050323 7239 raft_consensus.cc:1203] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 02:04:42.096819 7239 raft_consensus.cc:1203] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240503 02:04:42.157452 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:04:42.158874 19822 consensus_queue.cc:579] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.312s)
I20240503 02:04:42.159687 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e): 4838 bytes on disk
I20240503 02:04:42.160463 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(8ebe541161c84bd6b11bcd3d13d3bd2e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 02:04:42.208715 20210 ts_tablet_manager.cc:902] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 02:04:42.209363 20210 tablet_copy_client.cc:308] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 02:04:42.210014 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 66317a254eb644359f65610368d84d76 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:44053)
I20240503 02:04:42.210134 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 66317a254eb644359f65610368d84d76 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:44053: session id = efeb87c28ee04b3da5c686665b0c5c58-66317a254eb644359f65610368d84d76
I20240503 02:04:42.210986 7261 tablet_copy_source_session.cc:205] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:42.212100 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66317a254eb644359f65610368d84d76. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:04:42.212105 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 27c539b143674bdd8e3e5aaa4b78c1e9: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
I20240503 02:04:42.214071 20210 tablet_copy_client.cc:783] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:42.218993 20210 tablet_copy_client.cc:647] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.220520 20210 tablet_copy_client.cc:523] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.222003 20210 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:42.223935 20123 raft_consensus.cc:1050] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 02:04:42.224483 20123 consensus_queue.cc:237] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 02:04:42.226567 7239 raft_consensus.cc:1203] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.12->[2.13-2.13] Dedup: 2.13->[]
I20240503 02:04:42.226696 7371 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:42.227349 7500 raft_consensus.cc:1261] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:42.227754 20123 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:42.227874 20124 consensus_queue.cc:1035] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:42.230526 20126 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.231803 7500 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.232448 20210 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.232733 20210 tablet_bootstrap.cc:492] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:42.232908 20210 ts_tablet_manager.cc:1367] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 02:04:42.232753 7371 raft_consensus.cc:2932] T ec179e5e683d4ba099c459331ea5e578 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.233350 20210 raft_consensus.cc:348] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.233451 20210 raft_consensus.cc:729] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:42.233537 20210 consensus_queue.cc:260] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.233852 20210 ts_tablet_manager.cc:1398] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:04:42.234409 7094 catalog_manager.cc:5484] T ec179e5e683d4ba099c459331ea5e578 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 9 to 10, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
W20240503 02:04:42.234637 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.234851 20126 consensus_queue.cc:579] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c8ecb5fc8d804c9b9ca42ee4b2377f4e for more than 300 seconds (300.301s)
I20240503 02:04:42.236557 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-66317a254eb644359f65610368d84d76 received from {username='slave'} at 127.0.99.132:44053
I20240503 02:04:42.236640 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-66317a254eb644359f65610368d84d76 on tablet 66317a254eb644359f65610368d84d76 with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:42.242987 20135 consensus_queue.cc:579] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.271s)
I20240503 02:04:42.246224 7239 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
W20240503 02:04:42.247891 7177 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): Couldn't send request to peer bed387c0ff9241688cdf45f364dc6f5a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.248415 7500 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:04:42.248510 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.248817 20135 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:04:42.250696 20135 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:42.252327 7085 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 18468345c7924d4f88366524b56e8ab3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:42.252578 7500 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } } }
I20240503 02:04:42.253423 7099 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 6 to 7, NON_VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:42.274890 19822 consensus_queue.cc:579] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.209s)
I20240503 02:04:42.277601 7371 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.279052 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:04:42.279166 7306 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.279611 20263 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:04:42.279831 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.279902 20210 ts_tablet_manager.cc:902] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:42.280370 20210 tablet_copy_client.cc:308] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:42.280898 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 245cba4e5728491896e7995ca0d59b36 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:34781)
I20240503 02:04:42.281018 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 245cba4e5728491896e7995ca0d59b36 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:34781: session id = efeb87c28ee04b3da5c686665b0c5c58-245cba4e5728491896e7995ca0d59b36
I20240503 02:04:42.281505 19822 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.281955 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.282161 7523 tablet_copy_source_session.cc:205] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:42.283059 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245cba4e5728491896e7995ca0d59b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.284420 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ad24b47690b4263bdf4a9157db7c17f with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:42.284832 7094 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.285274 20210 tablet_copy_client.cc:783] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
W20240503 02:04:42.286204 7567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 189 similar messages]
I20240503 02:04:42.291289 20210 tablet_copy_client.cc:647] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.292492 20210 tablet_copy_client.cc:523] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.293684 20210 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:42.300871 20210 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.301121 20210 tablet_bootstrap.cc:492] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:42.301285 20210 ts_tablet_manager.cc:1367] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:04:42.301728 20210 raft_consensus.cc:348] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.301841 20210 raft_consensus.cc:729] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:42.302243 20210 consensus_queue.cc:260] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.302542 20210 ts_tablet_manager.cc:1398] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:04:42.302621 20362 raft_consensus.cc:1050] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 02:04:42.303124 20362 consensus_queue.cc:237] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 02:04:42.303716 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-245cba4e5728491896e7995ca0d59b36 received from {username='slave'} at 127.0.99.132:34781
I20240503 02:04:42.303802 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-245cba4e5728491896e7995ca0d59b36 on tablet 245cba4e5728491896e7995ca0d59b36 with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:42.305572 7633 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:04:42.305583 7502 raft_consensus.cc:1261] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240503 02:04:42.305706 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.305974 20263 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.306102 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.306267 20365 consensus_queue.cc:1035] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:04:42.308270 20263 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.308674 7500 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.310420 7633 raft_consensus.cc:2932] T 71f12a25a1db4650a6c37b5000a668c7 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.311270 7099 catalog_manager.cc:5484] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.326104 20021 raft_consensus.cc:1050] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 02:04:42.326615 20021 consensus_queue.cc:237] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 02:04:42.328408 7633 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:04:42.328507 7239 raft_consensus.cc:1261] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:04:42.328836 20369 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.328969 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.329085 20021 consensus_queue.cc:1035] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.329532 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.331120 20369 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.331589 7239 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.333300 7633 raft_consensus.cc:2932] T 527af716906b42729952db28e790bafd P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.334066 7098 catalog_manager.cc:5484] T 527af716906b42729952db28e790bafd P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.382861 20210 ts_tablet_manager.cc:902] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:42.383463 20210 tablet_copy_client.cc:308] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:42.384027 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet c18129344d124a5f8f688ec35ef69a55 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:34781)
I20240503 02:04:42.384157 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet c18129344d124a5f8f688ec35ef69a55 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:34781: session id = efeb87c28ee04b3da5c686665b0c5c58-c18129344d124a5f8f688ec35ef69a55
I20240503 02:04:42.385056 7523 tablet_copy_source_session.cc:205] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:42.385808 20210 ts_tablet_manager.cc:1875] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:42.387732 20210 ts_tablet_manager.cc:1888] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:04:42.387905 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18129344d124a5f8f688ec35ef69a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.388799 20210 tablet_copy_client.cc:783] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:42.393383 20210 tablet_copy_client.cc:647] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.394855 20210 tablet_copy_client.cc:523] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.396152 20210 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:42.402662 20210 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.402894 20210 tablet_bootstrap.cc:492] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:42.403045 20210 ts_tablet_manager.cc:1367] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:04:42.403417 20210 raft_consensus.cc:348] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.403528 20210 raft_consensus.cc:729] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:42.403673 20210 consensus_queue.cc:260] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.403931 20210 ts_tablet_manager.cc:1398] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240503 02:04:42.404441 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-c18129344d124a5f8f688ec35ef69a55 received from {username='slave'} at 127.0.99.132:34781
I20240503 02:04:42.404515 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-c18129344d124a5f8f688ec35ef69a55 on tablet c18129344d124a5f8f688ec35ef69a55 with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:42.412386 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36): 4838 bytes on disk
I20240503 02:04:42.412799 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(245cba4e5728491896e7995ca0d59b36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 02:04:42.413537 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76): 4838 bytes on disk
I20240503 02:04:42.413837 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(66317a254eb644359f65610368d84d76) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240503 02:04:42.414470 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55): 4583 bytes on disk
I20240503 02:04:42.414753 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(c18129344d124a5f8f688ec35ef69a55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
I20240503 02:04:42.448499 7633 raft_consensus.cc:1203] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 02:04:42.492849 20123 raft_consensus.cc:1050] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 02:04:42.493247 20123 consensus_queue.cc:237] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 02:04:42.494577 7500 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:42.495040 20126 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:42.495122 7239 raft_consensus.cc:1261] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 02:04:42.495318 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.495535 20126 consensus_queue.cc:1035] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:42.497289 20124 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.497819 7239 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.500236 7094 catalog_manager.cc:5484] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 12 to 13, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.501207 7500 raft_consensus.cc:2932] T f0539dded7554723b9e918e8cdfbab20 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.508126 20124 raft_consensus.cc:1050] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 5b46f0d4b665497b8d7e57bf4d5f36cb to VOTER
I20240503 02:04:42.508563 20124 consensus_queue.cc:237] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } }
I20240503 02:04:42.510005 7500 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:04:42.510334 7239 raft_consensus.cc:1261] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20240503 02:04:42.510721 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.510721 20125 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:04:42.510877 20126 consensus_queue.cc:1035] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.511157 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.512660 20125 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.513085 7500 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.515234 7094 catalog_manager.cc:5484] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.514425 7239 raft_consensus.cc:2932] T 81086ff737c84d9d9f04743d31c6151b P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { promote: false } } }
I20240503 02:04:42.521901 20126 consensus_queue.cc:579] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.588s)
I20240503 02:04:42.523658 20012 raft_consensus.cc:1050] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:42.524243 20012 consensus_queue.cc:237] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:42.525833 7633 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:04:42.526234 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.526253 7239 raft_consensus.cc:1261] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:42.526366 20362 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:04:42.526932 20362 consensus_queue.cc:1035] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:04:42.528548 20012 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:42.528990 7239 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:42.530400 7633 raft_consensus.cc:2932] T 0d00b8fcb29f47298643903ec0cba606 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:42.531409 7098 catalog_manager.cc:5484] T 0d00b8fcb29f47298643903ec0cba606 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
W20240503 02:04:42.603595 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:04:42.624226 20362 consensus_queue.cc:579] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer c778431b13784c7c81f3924b7f73e664 for more than 300 seconds (300.034s)
I20240503 02:04:42.643033 20012 raft_consensus.cc:1050] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 02:04:42.643469 20012 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 02:04:42.644997 7239 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240503 02:04:42.645113 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 151 similar messages]
I20240503 02:04:42.645243 7633 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:04:42.645372 20012 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:04:42.645787 20012 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.645951 7306 consensus_peers.cc:485] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.647938 20012 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.648731 7239 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.650502 7633 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.651223 7094 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.656311 7371 consensus_queue.cc:237] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 02:04:42.657428 7239 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 02:04:42.657603 7633 raft_consensus.cc:1261] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 02:04:42.657799 20012 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:04:42.658015 19822 consensus_queue.cc:1035] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:04:42.659325 20012 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.659772 7239 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.660590 7633 raft_consensus.cc:2932] T a0b1a4b2ede6493f9e4e094649b4b57d P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.661808 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0b1a4b2ede6493f9e4e094649b4b57d with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:42.662834 7098 catalog_manager.cc:5484] T a0b1a4b2ede6493f9e4e094649b4b57d P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 14 to 15, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.700641 7633 raft_consensus.cc:1203] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:04:42.724627 7632 raft_consensus.cc:1203] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:04:42.727792 20126 raft_consensus.cc:1050] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58: attempting to promote NON_VOTER 4356d1cb1050473a8d9e50ff09a24dc2 to VOTER
I20240503 02:04:42.728193 20126 consensus_queue.cc:237] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } }
I20240503 02:04:42.729547 7239 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 LEARNER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:04:42.729969 7371 raft_consensus.cc:1261] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Refusing update from remote peer efeb87c28ee04b3da5c686665b0c5c58: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:04:42.730324 19945 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:04:42.730525 20123 consensus_queue.cc:1035] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 02:04:42.731835 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.732560 19945 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.733042 7371 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.735307 7237 raft_consensus.cc:2932] T 68791b3adccd4168b55a2da4423232d7 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } } }
I20240503 02:04:42.735729 7094 catalog_manager.cc:5484] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 13 to 14, 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efeb87c28ee04b3da5c686665b0c5c58" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:42.739396 20210 ts_tablet_manager.cc:902] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181)
I20240503 02:04:42.739996 20210 tablet_copy_client.cc:308] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46181
I20240503 02:04:42.740422 7392 tablet_copy_service.cc:140] P bed387c0ff9241688cdf45f364dc6f5a: Received BeginTabletCopySession request for tablet 8ad24b47690b4263bdf4a9157db7c17f from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:51673)
I20240503 02:04:42.740530 7392 tablet_copy_service.cc:161] P bed387c0ff9241688cdf45f364dc6f5a: Beginning new tablet copy session on tablet 8ad24b47690b4263bdf4a9157db7c17f from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:51673: session id = efeb87c28ee04b3da5c686665b0c5c58-8ad24b47690b4263bdf4a9157db7c17f
I20240503 02:04:42.741494 7392 tablet_copy_source_session.cc:205] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:42.742142 20210 ts_tablet_manager.cc:1875] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:42.743963 20210 ts_tablet_manager.cc:1888] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:04:42.744123 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad24b47690b4263bdf4a9157db7c17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.744946 20210 tablet_copy_client.cc:783] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:42.749425 20210 tablet_copy_client.cc:647] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.750790 20210 tablet_copy_client.cc:523] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.752409 20210 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
W20240503 02:04:42.759351 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:04:42.761654 20210 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.761947 20210 tablet_bootstrap.cc:492] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:42.762144 20210 ts_tablet_manager.cc:1367] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240503 02:04:42.762581 20210 raft_consensus.cc:348] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.762692 20210 raft_consensus.cc:729] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:42.762789 20210 consensus_queue.cc:260] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.763083 20210 ts_tablet_manager.cc:1398] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:04:42.763621 7392 tablet_copy_service.cc:342] P bed387c0ff9241688cdf45f364dc6f5a: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-8ad24b47690b4263bdf4a9157db7c17f received from {username='slave'} at 127.0.99.132:51673
I20240503 02:04:42.763695 7392 tablet_copy_service.cc:433] P bed387c0ff9241688cdf45f364dc6f5a: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-8ad24b47690b4263bdf4a9157db7c17f on tablet 8ad24b47690b4263bdf4a9157db7c17f with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:42.784584 20439 ts_tablet_manager.cc:902] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Initiating tablet copy from peer 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469)
I20240503 02:04:42.785189 20439 tablet_copy_client.cc:308] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42469
I20240503 02:04:42.785902 7261 tablet_copy_service.cc:140] P 4356d1cb1050473a8d9e50ff09a24dc2: Received BeginTabletCopySession request for tablet 18468345c7924d4f88366524b56e8ab3 from peer bed387c0ff9241688cdf45f364dc6f5a ({username='slave'} at 127.0.99.130:39109)
I20240503 02:04:42.786007 7261 tablet_copy_service.cc:161] P 4356d1cb1050473a8d9e50ff09a24dc2: Beginning new tablet copy session on tablet 18468345c7924d4f88366524b56e8ab3 from peer bed387c0ff9241688cdf45f364dc6f5a at {username='slave'} at 127.0.99.130:39109: session id = bed387c0ff9241688cdf45f364dc6f5a-18468345c7924d4f88366524b56e8ab3
I20240503 02:04:42.786872 7261 tablet_copy_source_session.cc:205] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:42.787597 20439 ts_tablet_manager.cc:1875] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:42.789577 20439 ts_tablet_manager.cc:1888] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:04:42.789721 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18468345c7924d4f88366524b56e8ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.790625 20439 tablet_copy_client.cc:783] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:42.792405 7632 raft_consensus.cc:1203] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 02:04:42.795392 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:04:42.796233 20439 tablet_copy_client.cc:647] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:42.797461 20439 tablet_copy_client.cc:523] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:42.798882 20439 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap starting.
I20240503 02:04:42.806253 20439 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:42.806520 20439 tablet_bootstrap.cc:492] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Bootstrap complete.
I20240503 02:04:42.806718 20439 ts_tablet_manager.cc:1367] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 02:04:42.807087 20439 raft_consensus.cc:348] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:42.807193 20439 raft_consensus.cc:729] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bed387c0ff9241688cdf45f364dc6f5a, State: Initialized, Role: LEARNER
I20240503 02:04:42.807307 20439 consensus_queue.cc:260] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: true } }
I20240503 02:04:42.807634 20439 ts_tablet_manager.cc:1398] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.003s
I20240503 02:04:42.808184 7261 tablet_copy_service.cc:342] P 4356d1cb1050473a8d9e50ff09a24dc2: Request end of tablet copy session bed387c0ff9241688cdf45f364dc6f5a-18468345c7924d4f88366524b56e8ab3 received from {username='slave'} at 127.0.99.130:39109
I20240503 02:04:42.808260 7261 tablet_copy_service.cc:433] P 4356d1cb1050473a8d9e50ff09a24dc2: ending tablet copy session bed387c0ff9241688cdf45f364dc6f5a-18468345c7924d4f88366524b56e8ab3 on tablet 18468345c7924d4f88366524b56e8ab3 with peer bed387c0ff9241688cdf45f364dc6f5a
I20240503 02:04:42.878070 20021 consensus_queue.cc:579] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Leader has been unable to successfully communicate with peer 72577fce0313400ea862bf873738af54 for more than 300 seconds (300.052s)
I20240503 02:04:42.881181 7500 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } }
I20240503 02:04:42.882843 7237 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:42.882913 7371 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 02:04:42.883162 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.883450 20021 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:42.883625 20132 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:42.885552 20369 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 02:04:42.885906 7371 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 02:04:42.886008 7237 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } }
I20240503 02:04:42.887727 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40260acad62242c3a72197d2dafae187 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:42.888541 7099 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 9 to 10, VOTER 72577fce0313400ea862bf873738af54 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:42.891835 7500 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:42.893865 7237 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:42.893864 7371 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:04:42.893905 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.894253 20021 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:04:42.894426 7437 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123): Couldn't send request to peer efeb87c28ee04b3da5c686665b0c5c58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:42.894616 20021 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:04:42.896324 20369 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.897769 7086 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 40260acad62242c3a72197d2dafae187 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:04:42.897717 7237 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.897817 7371 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } } }
I20240503 02:04:42.900331 7094 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:42.917613 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f): 4923 bytes on disk
I20240503 02:04:42.918112 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(8ad24b47690b4263bdf4a9157db7c17f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 02:04:42.955299 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:43.009689 7419 maintenance_manager.cc:419] P bed387c0ff9241688cdf45f364dc6f5a: Scheduling UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3): 4583 bytes on disk
I20240503 02:04:43.010190 7309 maintenance_manager.cc:643] P bed387c0ff9241688cdf45f364dc6f5a: UndoDeltaBlockGCOp(18468345c7924d4f88366524b56e8ab3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 02:04:43.048602 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:43.049522 20021 raft_consensus.cc:1050] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:43.049903 20021 consensus_queue.cc:237] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.051199 7632 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:43.051380 7237 raft_consensus.cc:1261] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:04:43.051661 20369 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:04:43.051791 20448 consensus_queue.cc:1035] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:04:43.051904 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.053478 20448 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.053915 7632 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.055989 7237 raft_consensus.cc:2932] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.056843 7098 catalog_manager.cc:5484] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 10 to 11, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:43.063320 7088 maintenance_manager.cc:419] P b417ed481cd9481ba8418c1658ec8f19: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10880 bytes on disk
I20240503 02:04:43.065430 7087 maintenance_manager.cc:643] P b417ed481cd9481ba8418c1658ec8f19: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20240503 02:04:43.083016 20012 consensus_queue.cc:579] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Leader has been unable to successfully communicate with peer e648eca559ed4e5ab24a1b98755aa3eb for more than 300 seconds (300.493s)
W20240503 02:04:43.118623 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:04:43.216611 20021 raft_consensus.cc:1050] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:43.217069 20021 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.218573 7371 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.218585 7237 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.218766 7632 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.219139 20132 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:04:43.219291 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.219293 20021 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:43.219606 20494 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:43.221589 20448 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.222005 7632 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.222025 7371 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.224865 7237 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.224963 7099 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 reported cstate change: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
W20240503 02:04:43.225828 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 40260acad62242c3a72197d2dafae187: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:43.228858 7500 consensus_queue.cc:237] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.230293 7632 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.230338 7237 raft_consensus.cc:1261] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.230762 20494 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:43.231130 20132 consensus_queue.cc:1035] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:04:43.231402 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.231833 20132 raft_consensus.cc:1050] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:43.232271 20132 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.232650 20021 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
W20240503 02:04:43.234016 7436 consensus_peers.cc:485] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.234131 7633 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.234346 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 245cba4e5728491896e7995ca0d59b36 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:43.234968 20369 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:43.235311 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.235466 7091 catalog_manager.cc:5484] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:43.235323 7237 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.235950 20448 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:43.237287 7632 raft_consensus.cc:2932] T 245cba4e5728491896e7995ca0d59b36 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.239322 20494 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.240404 7632 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.241268 7350 tablet_service.cc:1514] Processing DeleteTablet for tablet 245cba4e5728491896e7995ca0d59b36 with delete_type TABLET_DATA_TOMBSTONED (TS bed387c0ff9241688cdf45f364dc6f5a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60094
I20240503 02:04:43.241374 7239 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.242372 20502 tablet_replica.cc:330] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: stopping tablet replica
I20240503 02:04:43.242497 20502 raft_consensus.cc:2229] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:43.242666 20502 raft_consensus.cc:2258] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:43.242846 7094 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:43.246368 7500 consensus_queue.cc:237] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.247013 20502 ts_tablet_manager.cc:1875] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:04:43.247447 7239 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.247615 7632 raft_consensus.cc:1261] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.247843 20021 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:43.248178 20021 consensus_queue.cc:1035] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:43.249330 20021 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.249693 7239 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.250877 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c18129344d124a5f8f688ec35ef69a55 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:43.251005 20502 ts_tablet_manager.cc:1888] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:04:43.251124 20502 log.cc:1196] T 245cba4e5728491896e7995ca0d59b36 P bed387c0ff9241688cdf45f364dc6f5a: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/245cba4e5728491896e7995ca0d59b36
I20240503 02:04:43.251880 7086 catalog_manager.cc:4830] TS bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181): tablet 245cba4e5728491896e7995ca0d59b36 (table rebalance_test_table_5 [id=1347ea81a2854d61b22e8e363d091380]) successfully deleted
I20240503 02:04:43.252171 7098 catalog_manager.cc:5484] T c18129344d124a5f8f688ec35ef69a55 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:04:43.253221 7632 raft_consensus.cc:2932] T c18129344d124a5f8f688ec35ef69a55 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.253650 7633 raft_consensus.cc:1203] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:04:43.273352 20508 ts_tablet_manager.cc:902] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Initiating tablet copy from peer 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921)
I20240503 02:04:43.273912 20508 tablet_copy_client.cc:308] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34921
I20240503 02:04:43.274479 7523 tablet_copy_service.cc:140] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Received BeginTabletCopySession request for tablet 40260acad62242c3a72197d2dafae187 from peer efeb87c28ee04b3da5c686665b0c5c58 ({username='slave'} at 127.0.99.132:34781)
I20240503 02:04:43.274609 7523 tablet_copy_service.cc:161] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Beginning new tablet copy session on tablet 40260acad62242c3a72197d2dafae187 from peer efeb87c28ee04b3da5c686665b0c5c58 at {username='slave'} at 127.0.99.132:34781: session id = efeb87c28ee04b3da5c686665b0c5c58-40260acad62242c3a72197d2dafae187
I20240503 02:04:43.275461 7523 tablet_copy_source_session.cc:205] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:04:43.276317 20508 ts_tablet_manager.cc:1875] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:04:43.278278 20508 ts_tablet_manager.cc:1888] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:04:43.278453 20508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40260acad62242c3a72197d2dafae187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:43.279685 20508 tablet_copy_client.cc:783] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 5 data blocks...
I20240503 02:04:43.284634 20508 tablet_copy_client.cc:647] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:43.285928 20508 tablet_copy_client.cc:523] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:43.287187 20508 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap starting.
I20240503 02:04:43.288651 20336 raft_consensus.cc:1050] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:43.289155 20336 consensus_queue.cc:237] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.290581 7632 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:43.290684 7500 raft_consensus.cc:1261] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:04:43.291077 20515 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:04:43.291201 20347 consensus_queue.cc:1035] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:04:43.291344 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.293169 20347 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.293596 7500 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.294878 7632 raft_consensus.cc:2932] T 66317a254eb644359f65610368d84d76 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.296439 7098 catalog_manager.cc:5484] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 9 to 10, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:43.300168 20508 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:43.300444 20508 tablet_bootstrap.cc:492] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Bootstrap complete.
I20240503 02:04:43.300621 20508 ts_tablet_manager.cc:1367] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240503 02:04:43.300999 20508 raft_consensus.cc:348] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:43.301098 20508 raft_consensus.cc:729] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efeb87c28ee04b3da5c686665b0c5c58, State: Initialized, Role: LEARNER
I20240503 02:04:43.301173 20508 consensus_queue.cc:260] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: true } }
I20240503 02:04:43.301378 20508 ts_tablet_manager.cc:1398] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:04:43.301854 7523 tablet_copy_service.cc:342] P 5b46f0d4b665497b8d7e57bf4d5f36cb: Request end of tablet copy session efeb87c28ee04b3da5c686665b0c5c58-40260acad62242c3a72197d2dafae187 received from {username='slave'} at 127.0.99.132:34781
I20240503 02:04:43.301924 7523 tablet_copy_service.cc:433] P 5b46f0d4b665497b8d7e57bf4d5f36cb: ending tablet copy session efeb87c28ee04b3da5c686665b0c5c58-40260acad62242c3a72197d2dafae187 on tablet 40260acad62242c3a72197d2dafae187 with peer efeb87c28ee04b3da5c686665b0c5c58
I20240503 02:04:43.363482 7371 raft_consensus.cc:1203] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:04:43.421940 7681 maintenance_manager.cc:419] P efeb87c28ee04b3da5c686665b0c5c58: Scheduling UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187): 4583 bytes on disk
I20240503 02:04:43.422420 7571 maintenance_manager.cc:643] P efeb87c28ee04b3da5c686665b0c5c58: UndoDeltaBlockGCOp(40260acad62242c3a72197d2dafae187) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 02:04:43.570827 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:43.576752 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:43.581161 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:43.639737 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:43.660137 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:04:43.740128 20362 raft_consensus.cc:1050] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:43.740525 20362 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.741932 7632 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:43.741967 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 02:04:43.742349 7305 consensus_peers.cc:485] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.742362 20362 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:43.742484 20012 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:43.744120 20362 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.744506 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.746119 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.748126 7091 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.751524 7371 consensus_queue.cc:237] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:43.752568 7632 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:43.752687 7239 raft_consensus.cc:1261] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer bed387c0ff9241688cdf45f364dc6f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:43.752982 19822 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:43.753106 20012 consensus_queue.cc:1035] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:43.754499 20362 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.754935 7239 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.755059 7632 raft_consensus.cc:2932] T 8ad24b47690b4263bdf4a9157db7c17f P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:43.755864 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ad24b47690b4263bdf4a9157db7c17f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:43.756690 7099 catalog_manager.cc:5484] T 8ad24b47690b4263bdf4a9157db7c17f P bed387c0ff9241688cdf45f364dc6f5a reported cstate change: config changed from index 12 to 13, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "bed387c0ff9241688cdf45f364dc6f5a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:43.800961 20336 raft_consensus.cc:1050] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2: attempting to promote NON_VOTER bed387c0ff9241688cdf45f364dc6f5a to VOTER
I20240503 02:04:43.801486 20336 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:43.802878 7371 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 LEARNER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.803292 20514 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:04:43.803835 7174 consensus_peers.cc:485] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:43.804039 7500 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:04:43.804370 20514 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:04:43.805999 20336 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.808040 7500 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.808106 7371 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.810745 7091 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 7 to 8, bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.814183 7239 consensus_queue.cc:237] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } }
I20240503 02:04:43.815487 7500 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.815488 7371 raft_consensus.cc:1261] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 4356d1cb1050473a8d9e50ff09a24dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:04:43.815883 20567 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:43.816022 20336 consensus_queue.cc:1035] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:04:43.817852 20336 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.819458 7085 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18468345c7924d4f88366524b56e8ab3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:43.819814 7500 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.820744 7371 raft_consensus.cc:2932] T 18468345c7924d4f88366524b56e8ab3 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } }
I20240503 02:04:43.821098 7099 catalog_manager.cc:5484] T 18468345c7924d4f88366524b56e8ab3 P 4356d1cb1050473a8d9e50ff09a24dc2 reported cstate change: config changed from index 8 to 9, VOTER e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:43.835750 7632 raft_consensus.cc:1203] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 02:04:43.909251 7438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 113 similar messages]
I20240503 02:04:44.027686 19822 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:04:44.027827 19822 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:44.028344 19822 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:44.028812 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:04:44.028970 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:04:44.029685 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:44.029954 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:44.030191 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:44.030364 19822 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:44.120599 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.142993 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:44.199460 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:44.204172 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:04:44.232662 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:04:44.247747 20580 raft_consensus.cc:1050] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb: attempting to promote NON_VOTER efeb87c28ee04b3da5c686665b0c5c58 to VOTER
I20240503 02:04:44.248250 20580 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:44.249825 7632 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 LEARNER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:44.249907 7239 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:44.250064 7371 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:04:44.250381 20613 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:44.250535 20580 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:04:44.250828 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:44.250869 20614 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:04:44.252632 20613 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.253057 7632 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.255987 7239 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.256582 7098 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:44.258198 7371 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } attrs { replace: true } } peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.259963 7500 consensus_queue.cc:237] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } }
I20240503 02:04:44.261716 7632 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:44.261888 7371 raft_consensus.cc:1261] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Refusing update from remote peer 5b46f0d4b665497b8d7e57bf4d5f36cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:04:44.262266 20614 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:04:44.262435 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:44.262619 20613 consensus_queue.cc:1035] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:04:44.263971 20369 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.265246 7632 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P efeb87c28ee04b3da5c686665b0c5c58 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.265837 7371 raft_consensus.cc:2932] T 40260acad62242c3a72197d2dafae187 P bed387c0ff9241688cdf45f364dc6f5a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } } }
I20240503 02:04:44.267212 7086 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40260acad62242c3a72197d2dafae187 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:44.267686 7091 catalog_manager.cc:5484] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb reported cstate change: config changed from index 12 to 13, VOTER 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8ecb5fc8d804c9b9ca42ee4b2377f4e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:04:44.270386 7221 tablet_service.cc:1514] Processing DeleteTablet for tablet 40260acad62242c3a72197d2dafae187 with delete_type TABLET_DATA_TOMBSTONED (TS 4356d1cb1050473a8d9e50ff09a24dc2 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35818
I20240503 02:04:44.270771 20618 tablet_replica.cc:330] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: stopping tablet replica
I20240503 02:04:44.270951 20618 raft_consensus.cc:2229] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:44.271220 20618 raft_consensus.cc:2258] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:44.271924 20618 ts_tablet_manager.cc:1875] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:04:44.275388 20618 ts_tablet_manager.cc:1888] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 02:04:44.275472 20618 log.cc:1196] T 40260acad62242c3a72197d2dafae187 P 4356d1cb1050473a8d9e50ff09a24dc2: Deleting WAL directory at /tmp/dist-test-task8Tfalf/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714701524113760-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/40260acad62242c3a72197d2dafae187
I20240503 02:04:44.276118 7085 catalog_manager.cc:4830] TS 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469): tablet 40260acad62242c3a72197d2dafae187 (table rebalance_test_table_3 [id=525842a82f914047b4e7a020aa11ca26]) successfully deleted
W20240503 02:04:44.303121 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:44.402477 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:44.421496 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:44.725205 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.859722 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.871922 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.908540 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.928367 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.942682 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.971241 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:44.995085 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:45.006440 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.040138 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:45.164495 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:45.174829 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.176054 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 02:04:45.223459 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.288750 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.448570 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 68791b3adccd4168b55a2da4423232d7: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:45.533362 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:45.540897 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:45.665347 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.708570 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:45.731127 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:45.979954 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:46.056985 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:46.168763 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:46.327047 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:46.346076 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:46.463665 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:46.591221 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:46.614648 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 02:04:46.627039 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:46.762812 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:04:46.766860 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:46.822515 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:46.959745 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a0b1a4b2ede6493f9e4e094649b4b57d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:47.029758 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:04:47.142814 20611 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:47.142938 20611 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:47.143333 20611 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bed387c0ff9241688cdf45f364dc6f5a (127.0.99.130:46181), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:47.143721 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "efeb87c28ee04b3da5c686665b0c5c58" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "bed387c0ff9241688cdf45f364dc6f5a" is_pre_election: true
I20240503 02:04:47.143905 7371 raft_consensus.cc:2396] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efeb87c28ee04b3da5c686665b0c5c58 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8.
W20240503 02:04:47.144387 7569 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:47.144461 7569 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efeb87c28ee04b3da5c686665b0c5c58; no voters: bed387c0ff9241688cdf45f364dc6f5a, c778431b13784c7c81f3924b7f73e664
W20240503 02:04:47.144620 7568 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:04:47.144598 20611 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:47.171896 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:47.349633 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.362502 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 02:04:47.391924 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.461915 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:47.510890 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.512387 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.587013 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.605746 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.647925 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:47.724056 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.735687 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 02:04:47.778904 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:47.790375 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.830384 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:47.855916 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.883702 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.886499 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:47.965313 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:48.042522 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6649e1feab5b40738003119a9122f07a: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:48.077088 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:48.115283 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:48.268797 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:48.611441 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:48.615738 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:48.666800 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:48.765005 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:48.918761 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:48.947978 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:04:48.985774 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:49.012827 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 02:04:49.174610 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:04:49.195437 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:49.251142 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 71f12a25a1db4650a6c37b5000a668c7: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:49.290875 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:49.305653 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:49.415835 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:49.685750 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:49.689656 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:49.872792 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:49.874689 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:49.949496 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:49.978778 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.080631 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:50.116189 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.158170 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.162945 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.182492 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:50.225333 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.235914 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.331199 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:50.380252 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:04:50.381253 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 27c539b143674bdd8e3e5aaa4b78c1e9: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:50.382128 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:50.443611 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.541173 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.684725 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:50.701527 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:50.822119 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:51.029088 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:51.081504 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:51.236109 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:51.353364 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:51.365216 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:51.522024 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:04:51.631785 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:04:51.661013 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c18129344d124a5f8f688ec35ef69a55: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:51.748505 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:51.846252 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:51.856139 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:51.859227 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:51.964948 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:52.102831 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.103336 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.181286 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.360752 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.379302 7568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 126 similar messages]
I20240503 02:04:52.418051 21319 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:52.418202 21319 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:04:52.418673 21319 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:04:52.419099 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:04:52.419287 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:04:52.419827 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:04:52.419986 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:52.420037 7438 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:04:52.420187 21319 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:52.486229 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.492517 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:52.496105 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:04:52.540426 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.584254 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:52.611574 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.678642 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.722433 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:52.740998 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 02:04:52.882412 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:52.882723 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:52.993095 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:53.034704 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:53.098609 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:53.102576 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:53.127018 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:53.384893 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:53.535584 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:53.544289 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:53.567615 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:53.575379 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:53.752082 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:53.874389 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:53.992626 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:54.028877 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:04:54.041088 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:04:54.095275 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:04:54.106990 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:54.151782 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:54.489392 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:54.525171 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:54.539867 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:54.699977 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:54.708573 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:54.915853 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:54.937656 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.072499 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:55.107198 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.119143 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.172189 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.243829 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:55.257434 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.419673 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:55.475451 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.498243 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:55.651527 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:55.664191 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:55.761477 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.805658 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:55.808897 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:56.007741 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:56.042003 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:56.093173 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:56.177459 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:56.197046 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:56.281328 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 02:04:56.367327 21253 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:04:56.367501 21253 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:04:56.368180 21253 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:04:56.368686 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:04:56.368856 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:04:56.369392 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:56.369558 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:04:56.369616 7307 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:04:56.369740 21253 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:56.379791 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:56.542778 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:56.617837 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:04:56.761430 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240503 02:04:56.762202 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:04:56.907240 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:56.959427 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:56.970176 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:56.970470 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:04:57.124168 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:57.151232 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.236533 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.404659 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 02:04:57.405328 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.462481 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.477820 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.523979 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.710529 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:04:57.710543 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:57.742787 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.743182 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 02:04:57.794706 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 68791b3adccd4168b55a2da4423232d7: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:57.863179 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:04:57.877252 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:57.936017 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:57.987162 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:58.067646 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:58.105109 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:58.178071 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:58.249111 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:58.315455 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:58.320546 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:58.343577 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:58.479497 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:58.569137 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:04:58.636226 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:58.664314 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:04:58.960496 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:59.054114 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:04:59.126487 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:04:59.150341 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:04:59.288940 7084 catalog_manager.cc:4573] TS 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a0b1a4b2ede6493f9e4e094649b4b57d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
W20240503 02:04:59.300724 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:59.328900 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:04:59.495307 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:59.556773 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:04:59.559080 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:04:59.680833 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:04:59.714063 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:04:59.793965 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:04:59.983392 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:04:59.989986 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.010579 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.113559 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.176162 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:00.197655 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.205672 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:00.319388 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:00.379825 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.386431 7083 catalog_manager.cc:4573] TS e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8ad24b47690b4263bdf4a9157db7c17f: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:05:00.427210 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:00.577469 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.612112 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:00.615041 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.618544 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:00.657342 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.727905 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:00.922077 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:00.989055 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:01.021476 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:01.034340 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:01.202098 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:01.502940 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:01.733590 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 02:05:01.882500 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:01.899981 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:01.910128 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:05:01.993914 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.170562 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.242703 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.299741 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.382123 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.401033 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.432170 7569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 02:05:02.445227 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:05:02.452193 21188 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:02.452308 21188 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:05:02.452906 21188 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:05:02.453222 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:05:02.453413 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:05:02.454088 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:05:02.454180 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:05:02.454257 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:05:02.454384 21188 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:02.498931 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.604542 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.642127 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.648346 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.683332 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:02.713953 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.765468 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.783428 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 02:05:02.835237 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:02.857909 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.884402 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:02.922340 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:03.098006 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:03.135671 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:05:03.136426 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:03.225363 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:03.227964 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:03.377254 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:03.516309 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:03.627877 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:03.725283 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:03.811394 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:03.832152 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:04.188459 7436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:05:04.420025 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:05:04.472795 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:04.479226 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:04.496775 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:05:04.549881 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:04.563187 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:04.610483 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:04.673539 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:04.723693 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:04.819726 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:04.822530 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:04.983359 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:04.990756 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:05.079041 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.183176 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.187942 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.304147 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.304124 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.316015 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.460532 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.473315 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:05.560269 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.615322 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:05.624095 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:05.632261 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:05.700044 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:05.849371 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:05.890113 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:05.995152 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:06.139145 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:06.154314 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 02:05:06.180424 22689 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:05:06.180575 22689 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:05:06.181089 22689 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:05:06.181501 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:05:06.181669 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:05:06.182183 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:05:06.182296 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:05:06.182356 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:05:06.182500 22689 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:06.410744 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 02:05:06.695585 22734 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:06.695755 22734 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:05:06.696303 22734 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b46f0d4b665497b8d7e57bf4d5f36cb (127.0.99.131:34921), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:05:06.696659 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" is_pre_election: true
I20240503 02:05:06.696944 7500 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356d1cb1050473a8d9e50ff09a24dc2 in term 2.
W20240503 02:05:06.697503 7174 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
W20240503 02:05:06.697850 7176 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
I20240503 02:05:06.697921 7176 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:05:06.698180 22734 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:06.804651 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:05:06.842059 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:06.922453 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:07.030493 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:05:07.122870 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.133989 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.143091 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:07.188835 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.296794 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.339771 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:07.345742 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:07.519495 7567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 02:05:07.588384 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.607206 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:07.616390 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.678035 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:07.692826 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.725442 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:07.812565 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 02:05:07.837123 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:07.927242 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:07.980051 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:08.033771 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.071074 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.078893 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.082298 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.083010 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.119925 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:08.334095 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:08.373468 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:08.524114 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:08.546622 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:08.667845 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:05:09.011713 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:09.100060 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:09.331580 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:09.352794 7437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:05:09.478368 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 02:05:09.494618 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:09.549369 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:09.670940 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:05:09.738559 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:09.767077 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:09.785017 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:09.851461 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:09.904995 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:10.000320 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.084089 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.113638 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:10.164094 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:10.176358 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.180662 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:10.290539 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.365849 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.370795 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:10.432109 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:10.495023 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.514897 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:10.576212 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.595472 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.637589 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.732028 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.759445 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:10.991508 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:11.119040 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:11.206156 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:11.256414 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:11.470216 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:11.759089 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:11.993896 7439 consensus_peers.cc:485] T 40260acad62242c3a72197d2dafae187 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:05:12.004145 7308 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:12.155429 7569 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:12.190506 7437 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.258020 7308 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 02:05:12.265246 7305 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.406837 7307 consensus_peers.cc:485] T 71f12a25a1db4650a6c37b5000a668c7 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.542326 7570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 02:05:12.542901 7570 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 02:05:12.549122 7436 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:12.591190 7438 consensus_peers.cc:485] T 527af716906b42729952db28e790bafd P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.595891 7569 consensus_peers.cc:485] T b4628afbfeff4f8bbaba5da1e89e4850 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.623463 7307 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:12.637012 7568 consensus_peers.cc:485] T ec179e5e683d4ba099c459331ea5e578 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.714810 7306 consensus_peers.cc:485] T 7d332a7c255b44589d54b1f583a2f9c4 P bed387c0ff9241688cdf45f364dc6f5a -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:12.722765 7569 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 02:05:12.814180 23266 raft_consensus.cc:482] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:12.814339 23266 raft_consensus.cc:504] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34921 } attrs { replace: true } } peers { permanent_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42469 } } peers { permanent_uuid: "e648eca559ed4e5ab24a1b98755aa3eb" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } }
I20240503 02:05:12.815034 23266 leader_election.cc:288] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4356d1cb1050473a8d9e50ff09a24dc2 (127.0.99.129:42469), e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981)
I20240503 02:05:12.815716 7239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb00fcf563b74243b7c955ddacf1d6cf" candidate_uuid: "5b46f0d4b665497b8d7e57bf4d5f36cb" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4356d1cb1050473a8d9e50ff09a24dc2" is_pre_election: true
I20240503 02:05:12.815881 7239 raft_consensus.cc:2454] T fb00fcf563b74243b7c955ddacf1d6cf P 4356d1cb1050473a8d9e50ff09a24dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b46f0d4b665497b8d7e57bf4d5f36cb in term 2.
W20240503 02:05:12.816095 7438 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:05:12.816191 7436 leader_election.cc:334] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111)
I20240503 02:05:12.816272 7436 leader_election.cc:302] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4356d1cb1050473a8d9e50ff09a24dc2, 5b46f0d4b665497b8d7e57bf4d5f36cb; no voters: c778431b13784c7c81f3924b7f73e664, e648eca559ed4e5ab24a1b98755aa3eb
I20240503 02:05:12.816394 23266 raft_consensus.cc:2726] T fb00fcf563b74243b7c955ddacf1d6cf P 5b46f0d4b665497b8d7e57bf4d5f36cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:12.817343 7305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 02:05:12.841209 7436 consensus_peers.cc:485] T 8ebe541161c84bd6b11bcd3d13d3bd2e P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.894789 7570 consensus_peers.cc:485] T 68791b3adccd4168b55a2da4423232d7 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:12.927959 7569 consensus_peers.cc:485] T f0539dded7554723b9e918e8cdfbab20 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.039585 7176 consensus_peers.cc:485] T 66317a254eb644359f65610368d84d76 P 4356d1cb1050473a8d9e50ff09a24dc2 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.053514 7437 consensus_peers.cc:485] T 3b15d48b2fa04269b98c6f801f54df4f P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:13.150193 7307 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:13.154079 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:05:13.154810 7086 catalog_manager.cc:4573] TS c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ba04b45ae1343f3aa4e5a82bbf7f486: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111)
W20240503 02:05:13.159857 7570 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:13.179841 7569 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.225015 7305 consensus_peers.cc:485] T 0d00b8fcb29f47298643903ec0cba606 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.299698 7567 consensus_peers.cc:485] T 81086ff737c84d9d9f04743d31c6151b P efeb87c28ee04b3da5c686665b0c5c58 -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.379943 7570 consensus_peers.cc:485] T c95d3c29bfbd4899b27b770bf9d3f4a6 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c8ecb5fc8d804c9b9ca42ee4b2377f4e (127.0.99.134:43615): Couldn't send request to peer c8ecb5fc8d804c9b9ca42ee4b2377f4e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43615: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 02:05:13.421690 7437 consensus_peers.cc:485] T 245cba4e5728491896e7995ca0d59b36 P 5b46f0d4b665497b8d7e57bf4d5f36cb -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn't send request to peer 72577fce0313400ea862bf873738af54. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:05:13.445039 7305 consensus_peers.cc:485] T 5f35ccb9c21b4c10b9187596d2f6929c P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:05:13.552013 7307 consensus_peers.cc:485] T a432cd5fe20941d6bc73dae386d85170 P bed387c0ff9241688cdf45f364dc6f5a -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 02:05:13.587131 7569 consensus_peers.cc:485] T cc68f31e6ef8411e96d46b2419bf9676 P efeb87c28ee04b3da5c686665b0c5c58 -> Peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Couldn't send request to peer c778431b13784c7c81f3924b7f73e664. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 02:05:13.652057 22886 raft_consensus.cc:482] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c778431b13784c7c81f3924b7f73e664)
I20240503 02:05:13.652176 22886 raft_consensus.cc:504] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efeb87c28ee04b3da5c686665b0c5c58" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33123 } } peers { permanent_uuid: "bed387c0ff9241688cdf45f364dc6f5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "c778431b13784c7c81f3924b7f73e664" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "72577fce0313400ea862bf873738af54" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33955 } attrs { promote: false } }
I20240503 02:05:13.652689 22886 leader_election.cc:288] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efeb87c28ee04b3da5c686665b0c5c58 (127.0.99.132:33123), c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981), 72577fce0313400ea862bf873738af54 (127.0.99.136:33955)
I20240503 02:05:13.653009 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9796308eacd94fb5ad3cb524aa590cfa" candidate_uuid: "bed387c0ff9241688cdf45f364dc6f5a" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "efeb87c28ee04b3da5c686665b0c5c58" is_pre_election: true
I20240503 02:05:13.653159 7632 raft_consensus.cc:2454] T 9796308eacd94fb5ad3cb524aa590cfa P efeb87c28ee04b3da5c686665b0c5c58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed387c0ff9241688cdf45f364dc6f5a in term 2.
W20240503 02:05:13.653661 7307 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c778431b13784c7c81f3924b7f73e664 (127.0.99.135:43981): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111)
W20240503 02:05:13.653991 7306 leader_election.cc:334] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Network error: Client connection negotiation failed: client connection to 127.0.99.136:33955: connect: Connection refused (error 111)
I20240503 02:05:13.654070 7306 leader_election.cc:302] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: bed387c0ff9241688cdf45f364dc6f5a, efeb87c28ee04b3da5c686665b0c5c58; no voters: 72577fce0313400ea862bf873738af54, c778431b13784c7c81f3924b7f73e664
I20240503 02:05:13.654222 22886 raft_consensus.cc:2726] T 9796308eacd94fb5ad3cb524aa590cfa P bed387c0ff9241688cdf45f364dc6f5a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:13.782196 7305 consensus_peers.cc:485] T 97ffb59d3ab64d03aca5d30a7c3be1e1 P bed387c0ff9241688cdf45f364dc6f5a -> Peer e648eca559ed4e5ab24a1b98755aa3eb (127.0.99.133:34145): Couldn't send request to peer e648eca559ed4e5ab24a1b98755aa3eb. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:34145: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 02:05:13.950745 7176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43981: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:05:14.170864 7568 consensus_peers.cc:485] T be1f4d2622c6452b869f88a301a5e9bc P efeb87c28ee04b3da5c686665b0c5c58 -> Peer 72577fce0313400ea862bf873738af54 (127.0.99.136:33955): Couldn'